Consider, there are two classes of data and we have learned the SVM parameters in terms Lagrange multipliers. There are many learning techniques to learn these parameters quadratic programming or sequential minimal optimization etc.   The question is, once we have learned our classifier, can we update the parameters somehow, once we get a new unknown sample . We can easily pass the unknown sample through the classifier and get a result.     But is there a way to  learn something blindly from the sample?  This is essential for our case: in our application the input to the classifier  changes slowly over time. So we need to capture the slow changes somehow blindly. Is it possible?    The list of answers from  are related to how to train when you get a new sample. Kindly note in that case you know the label of the new sample, in other words the class of the new sample is known or using oracle. The question here refers to the case when the class of the sample is not known As such in my opinion this is not duplicate. This question may not have an answer though. The closet approach so far seen is  active learning, however this not answer the original question of slow changing. To be more specific some active learning considers support vectors and weights as a metric to decide a label but here question is related to change of support vectors to cope with slow changes.  Let me write other way, I am looking for active learning without knowing the label. All those references deal with how to choose a sample, the oracle or somebody would specify the class of the sample.
121708
I'm using libsvm and I noticed that everytime I call svmtrain(), I create a new model and that there seems to be no option to put data in an existing model. Is this possible to do however? Am I just not seeing this aspect in libsvm?
99247
Is there an existing methodology for applying a supervised learning model to a uncertain dataset? For example, say we have a dataset with classes A and B:  +----------+----------+-------+-----------+ | FeatureA | FeatureB | Label | Certainty | +----------+----------+-------+-----------+ |        2 |        3 | A     | 50%       | |        3 |        1 | B     | 80%       | |        1 |        1 | A     | 100%      | +----------+----------+-------+-----------+   How could we train a machine learning model on this? Thanks.
102428
In the RMSProp algorithm (And similar algorithms) that are used in Machine Learning in the subject of Adaptive Learning Rates, the squares of the gradients are used in the algorithm step.  Is there motivation for why the squares of the gradients are used?  Thanks!
75895
A paper "Accurately computing running variance" at  shows how to compute running mean, variance and standard deviations.  Are there algorithms where the parameters of a linear or logistic regression model can be similarly "dynamically" updated as each new training record is provided?
49712
I have a classification problem with 2 classes (positive and negative). Usually, in such classification problems, all the samples will be labelled either 'positive' or 'negative'. In my dataset, some of the samples possess a combination of both positive and negative characteristics. Formally, if the dataset is $x$, then,      $x=x_1 \cup x_2 \cup x_3$   where $x_1$ is the set of all positive samples, $x_2$ is the set of all negative samples and $x_3$ is the set of samples that contain the characteristics of both the classes. As far as I could think of, this situation could be handled in 2 ways,   Ignore $x_3$ (samples that contain characteristics of both classes) and treat the problem as a traditional binary classification. Label the samples in $x_3$ with both labels (positive &amp; negative) and consider this as a multi-label classification problem.   I wish to follow the second option, as it is more natural choice. The reason being, ignoring some samples from the dataset gives me a feeling of manipulating the dataset artificially which may affect the performance of the classifier in the real world scenario. In this context, I have the following questions,   Is it correct to treat this a multi-label classification problem. If so, is the intuition which is explained in the previous paragraph correct? Is there any other learning paradigm that can handle this scenario? If so, please provide reference to relevant literature.
49384
I'm using Python Keras package for neural network. This is the . Is batch_size equals to number of test samples? From Wikipedia we have  information:     However, in other cases, evaluating the sum-gradient may require   expensive evaluations of the gradients from all summand functions.   When the training set is enormous and no simple formulas exist,   evaluating the sums of gradients becomes very expensive, because   evaluating the gradient requires evaluating all the summand functions'   gradients. To economize on the computational cost at every iteration,   stochastic gradient descent samples a subset of summand functions at   every step. This is very effective in the case of large-scale machine   learning problems.   Above information is describing test data? Is this same as batch_size in keras (Number of samples per gradient update)?
123350
There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few:    @DikranMarsupial's  to    However, the answers to those threads are fairly generic and mostly highlight the issues with particular approaches to cross validation and model selection.   To make things as concrete as possible, say for example that we are working with an SVM with an RBF kernel:  $K(x, x' ) = (\gamma \, \vert x - x'\vert)^2$, and that I have a dataset of features X and labels y, and that I want to    Find the best possible values of my model ($\gamma$ and $C$) Train the SVM with my dataset (for final deployment) Estimate the generalization error and the uncertainty (variance) around this error   To do so, I would personally do  a grid search, e.g. I try every possible combination of  $C$ and $\gamma$. For simplicity, we can assume the following ranges:   $C \in \{10, 100, 1000\}$ $\gamma \in \{0.1, 0.2, 0.5, 1.0\}$   More specifically, using my full dataset I do the following:   For every ($C$,$\gamma$) pair, I do repeated iterations (e.g. 100 random repetitions) of $K$-fold cross validation (e.g. $K=10$), on my dataset, i.e. I train my SVM on $K-1$ folds and evaluate the error on the fold left, iterating through all $K$ folds. Overall, I collect 100 x 10 = 1000 test errors. For each such ($C$,$\gamma$) pair, I compute the mean and the variance of those 1000 test errors $\mu_M, \sigma_M$.   Now I want to choose the best model (the best kernel parameters) that I would use to train my final SVM on the full dataset. My understanding is that choosing the model that had the lowest error mean and variance $\mu_M$ and $\sigma_M$ would be the right choice, and that this model's $\mu_M$ are $\sigma_M$ are my best estimates of the model's generalization error bias and variance when training with the full dataset.  BUT, after reading the answers in the threads above, I am getting the impression that this method for choosing the best SVM for deployment and/or for estimating its error (generalization performance), is flawed, and that there are better ways of choosing the best SVM and reporting its error. If so, what are they? I am looking for a concrete answer please.  Sticking to this problem, how specifically can I choose the best model and properly estimate its generalization error?
118172
I am learning how to use libsvm through sklearn.svm in python. I read  about what happens and why when you change the C value as part of your model. My intuition from what I've learned would be that lower C values would use less support vectors to make a more general classification, while higher C values would use more support vectors to attempt to 'overfit' and account for all outliers.   That is not the case. For an example where I looped through a set of C values like so:  print(c) model = svm.SVC(kernel='linear', C=c) model.fit(Xtrain, ytrain) print("support vectors:", len(model.support_))   I got results:  1.0 support vectors: 1810 10.0 support vectors: 1750 100.0 support vectors: 1626 1000.0 support vectors: 1558   As you can see, as C goes up, the number of support vectors used in the model goes down. Why does this happen?
13209
I am working with many algorithms: RandomForest, DecisionTrees, NaiveBayes, SVM (kernel=linear and rbf), KNN, LDA and XGBoost. All of them were pretty fast except for SVM. That is when I got to know that it needs feature scaling to work faster. Then I started wondering if I should do the same for the other algorithms.
53185
Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of epochs and got the optimal epoch number as a result.  Now you want to train the network for the optimal number of epochs on the whole data used for CV. In my opinion following problem arises: When using a constant mini batch size, number of iterations varies for datasets of different size - so your optimal epoch number found for the smaller training split in the CV can not be applied to training with the whole CV data. Am I missing something here? Right at the moment I can not find a proper solution for finding the optimal training epochs for my NN. I searched the net for some time now, but could not find an answer.  P.S. I only described the problem after the inner loop of nested CV, but the same problem arises after the outer loop (training the model on ALL data, not only on CV data for putting the model to production).
52981
Does the cross-entropy cost make sense in the context of regression (as opposed to classification)? If so, could you give a toy example through TensorFlow? If not, why not?   I was reading about cross-entropy in  by Michael Nielsen and it seems like something that could naturally be used for regression as well as classification, but I don't understand how you'd apply it efficiently in TensorFlow since the loss functions take logits (which I don't really understand either) and they're listed under classification
94552
I am currently reading Bishop [1] and got confusion on why should we take IRLS (Iterative Re-weighted Least Square) as it seems that using gradient descent that with one derivative at a time would solve the problem, what is the meaning of introducing Hessian matrix? Or did I understand anywhere wrong with that algorithm?   Could it be faster? Why? What does "re-weighted" mean here? Isn't that gradient descent also updates their weight iteratively so the weights are also "re-weighted"? The method that IRLS takes is Newton-Raphson, which could give exactly the same result with standard least square solution in linear regression model as below.   $$ w_{new}\; =\; w_{old}\; -\; H^{-1}\nabla E(w) $$  $$ \nabla E(w) = \sum_{n\; =\; 1}^{N}{\left( w^{T}\phi _{n}-t_{n} \right)}\phi _{n}\; =\; \phi ^{T}\phi w\; -\; \phi ^{T}t $$  $$ \nabla\nabla E(w) = \sum_{n\; =\; 1}^{N}{\phi _{n}}\phi _{n}^{T}\; =\; \phi ^{T}\phi  $$  $$ w_{new}\; =\; w_{old}\; -\; \left( \phi ^{T}\phi  \right)^{-1}\left\{ \phi ^{T}\phi w_{old}\; -\; \phi ^{T}t \right\}\; =\; \left( \phi ^{T}\phi  \right)^{-1}\phi ^{T}t $$  Therefore I just wonder, when the method once applied in linear regression, is there only one iteration that the algorithm is required to run?  [1]: Bishop, Christopher (2006), Pattern Recognition and Machine Learning, Springer-Verlag New York
104906
I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
101892
I've got this problem where basically:  $$ Z = f(X) \sim Normal(0, 1) $$  $f$ is pretty non-linear and I don't have its inverse function. In practice (using Stan's MCMC samplers or Tensorflow's optimisers or by using literally any method), is it possible to draw samples from X?    My limited progress:  I don't really know how I'd specify such a model in Stan, or if it's even possible ($f$ is non-linear, almost certainly non-monotonic so I can't add a Jacobian adjustment and treat X as a parameter).  I dunno if treating X as a variable and constructing an objective that applies $f$ to X and optimises in the direction that makes the sample "more normal" is the way to go here. I don't think that such an approach would scale well, with increasing dimensionality of X and Z.  I'm looking for any other ideas on how to sample from X. Is it possible to use the KL divergence here somehow?
77287
Let's say you have a dataset generated from real world sampling which has lots of duplicates (the dependent and independent variables are identical) and you want to train a classifier to predict the dependent variable in future samples from the real world. If you were to take a train test split from this dataset you'd inevitably have records which appear in both the train and the test dataset, so to avoid this you would deduplicate the original before doing a train test split.   My question is - is it the right thing to deduplicate and train on deduplicated data and test on totally unseen data, or is it the right thing to train and test from the original distribution coming from the real world, and accept that we would do well on previously seen examples anyway?
104639
I searched on topics of the bias and variance trade-off and got back lots of questions with different levels of response. The information is scattering too much and unsystematic to answer my own question which stuck me. So I open a new question here and would like to hackle how I generally run a machine learning modelling process and ask for help on the issue of bias-variance assessment where I was confused or had no idea. If anyone noticed any useful posts to my questions please help point out the links and many thanks.  I execute the process basically according to the instruction below:    Question 1: When I start at step one, I try to think an issue that all my following analysis will be based on this train-test split. How about another split and is it possible to cause an obvious change in analysis results (final models, performance metrics, important features etc.)? If so, that means the analysis was split-dependent.   I have some thoughts here: assuming there exists an unknown underlying distribution of the domain set X (the whole dataset except the target class y), if the sample distribution of training set S was approximately similar to that of test set T, then the learning process should reach a good performance. Otherwise, if the two sample distributions differed in some non-trivial way, then the learner/classifier from training set S may not probably do a good job on test set T.     I made a trial to execute a 10-fold cross validation on the whole dataset and to see how variable the ROC AUC score was. In my particular dataset, I got a set of 10 scores with large variation (from maximum 0.85 to minimum 0.59). This led to my worry whether the current split I adopted for analysis was poor one.  To expand Question 1, I would like to ask further: what is the topic of this issue in machine learning theory? Is it the model stability or model variance?  I tuned the hyper-parameter/parameters in step 2 via cross validation. I try to use validation curve (refer to ) as below to choose the parameter value and avoid under/over-fitting problems.     Question 2: If this curve worked well in detecting under/over-fitting problems, what metric should be used: accuracy score/error rate, presicion, recall, f1 or roc_auc? (My current project was a classification problem with an imbalanced binary target class.)  According to the definition of bias, average error rate could be the measure of bias, so accuracy score/error rate seemed more reasonable in bias/variance issue. In fact, I concerned roc_auc score more than accuracy itself.   Question 3: how can we measure the variance? I saw a plot  and the gap was said to be able to indicate the potential variance (poorer performance on validation set than that on training set). Any other (better) way/metric can be used for this point?    In my analysis, the bias/variance check was run in the parameter tuning stage which seemed to return me a candidate model with a balanced bias/variance combination.  Question 4: does my bias/variance analysis miss some steps and how do I adjust it?  many ensemble learning methods have their own way to reduce variance or bias and  I hope to objectively evaluate the bias/variance for each candidate model.   Thanks for reading my long question but it really help for me to deliver a professional practice in my daily work. Many thanks again.
10931
Given the data points $x_1, \ldots, x_n \in \mathbb{R}^d$ and labels $y_1, \ldots, y_n \in \left \{-1, 1 \right\}$, the hard margin SVM primal problem is  $$ \text{minimize}_{w, w_0} \quad \frac{1}{2} w^T w $$ $$ \text{s.t.} \quad  \forall i: y_i (w^T x_i + w_0) \ge 1$$  which is a quadratic program with $d+1$ variables to be optimized for and $i$ constraints. The dual  $$ \text{maximize}_{\alpha} \quad \sum_{i=1}^{n}{\alpha_i} - \frac{1}{2}\sum_{i=1}^{n}{\sum_{j=1}^{n}{y_i y_j \alpha_i \alpha_j x_i^T x_j}}$$ $$ \text{s.t.} \quad \forall i: \alpha_i \ge 0 \land \sum_{i=1}^{n}{y_i \alpha_i} = 0$$ is a quadratic program with $n + 1$ variables to be optimized for and $n$ inequality and $n$ equality constraints.  When implementing a hard margin SVM, why would I solve the dual problem instead of the primal problem? The primal problem looks more 'intuitive' to me, and I don't need to concern myself with the duality gap, the Kuhn-Tucker condition etc.  It would make sense to me to solve the dual problem if $d \gg n$, but I suspect there are better reasons. Is this the case?