I have a dataset of roughly 1000 subjects' data and their labels. The ultimate goal is to build a neural network which can classify the data as either 0 or 1. My current strategy is to do 10-fold CV, wherein I take a different 10% of the data for testing while training on the remaining 90% (repeated many times). There are many hyperparameters (hidden layer size, net type, etc) which I am optimizing based on the average accuracy of the 10-folds. So at the end of this, I've obtained the best hyperparameters to get the highest average accuracy.   When it is time to actually use this classifier on new data, which model do I use? If my training set is now those 1000 datapoints, then my previously optimized hyperparameters are no longer optimal for that training set.  I may have a fundamental misunderstanding of k-fold, but if not then how do I go about doing this (designing a classifier with labels to be used on unlabeled data in the future)?
123350
There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few:    @DikranMarsupial's  to    However, the answers to those threads are fairly generic and mostly highlight the issues with particular approaches to cross validation and model selection.   To make things as concrete as possible, say for example that we are working with an SVM with an RBF kernel:  $K(x, x' ) = (\gamma \, \vert x - x'\vert)^2$, and that I have a dataset of features X and labels y, and that I want to    Find the best possible values of my model ($\gamma$ and $C$) Train the SVM with my dataset (for final deployment) Estimate the generalization error and the uncertainty (variance) around this error   To do so, I would personally do  a grid search, e.g. I try every possible combination of  $C$ and $\gamma$. For simplicity, we can assume the following ranges:   $C \in \{10, 100, 1000\}$ $\gamma \in \{0.1, 0.2, 0.5, 1.0\}$   More specifically, using my full dataset I do the following:   For every ($C$,$\gamma$) pair, I do repeated iterations (e.g. 100 random repetitions) of $K$-fold cross validation (e.g. $K=10$), on my dataset, i.e. I train my SVM on $K-1$ folds and evaluate the error on the fold left, iterating through all $K$ folds. Overall, I collect 100 x 10 = 1000 test errors. For each such ($C$,$\gamma$) pair, I compute the mean and the variance of those 1000 test errors $\mu_M, \sigma_M$.   Now I want to choose the best model (the best kernel parameters) that I would use to train my final SVM on the full dataset. My understanding is that choosing the model that had the lowest error mean and variance $\mu_M$ and $\sigma_M$ would be the right choice, and that this model's $\mu_M$ are $\sigma_M$ are my best estimates of the model's generalization error bias and variance when training with the full dataset.  BUT, after reading the answers in the threads above, I am getting the impression that this method for choosing the best SVM for deployment and/or for estimating its error (generalization performance), is flawed, and that there are better ways of choosing the best SVM and reporting its error. If so, what are they? I am looking for a concrete answer please.  Sticking to this problem, how specifically can I choose the best model and properly estimate its generalization error?
37915
To use SVM or Neural Network it needs to transform (encode) categorical variables into numeric variables, the normal method in this case is to use 0-1 binary values with the k-th categorical value transformed to be (0,0,...,1,0,...0) (1 is on the k-th position). Is there other methods to do this, especially when there are a large number of categorical values(e.g.10000) such that the 0-1 representation will introduce a large number of additional dimensions(input units) in Neural Network which seems not quite desired or expected?   I am asking about general strategies.
81656
How can one use nested cross validation for model selection?   From what I read online, nested CV works as follows:   There is the inner CV loop, where we may conduct a grid search (e.g. running K-fold for every available model, e.g. combination of hyperparameters/features) There is the outer CV loop, where we measure the performance of the model that won in the inner fold, on a separate external fold.    At the end of this process we end up with $K$ models ($K$ being the number of folds in the outer loop). These models are the ones that won in the grid search within the inner CV, and they are likely different (e.g. SVMs with different kernels, trained with possibly different features, depending on the grid search).  How do I choose a model from this output? It looks to me that selecting the best model out of those $K$ winning models would not be a fair comparison since each model was trained and tested on different parts of the dataset.  So how can I use nested  CV for model selection?   Also I have read threads discussing how nested model selection is useful for analyzing the learning procedure. What types of analysis /checks can I do with the scores that I get from the outer K folds?
75679
I have some data and I want to build a model (say a linear regression model) out of this data. In a next step, I want to apply Leave-One-Out Cross-Validation (LOOCV) on the model so see how good it performs.  If I understood LOOCV right, I build a new model for each of my samples (the test set) using every sample except this sample (the training set). Then I use the model to predict the test set and calculate the errors $(\text{predicted} - \text{actual})$.  In a next step I aggregate all the errors generated using a chosen function, for example mean squared error. I can use these values to judge on the quality (or goodness of fit) of the model.  Question: Which model is the model these quality-values apply for, so which model should I choose if I find the metrics generated from LOOCV appropriate for my case? LOOCV looked at $n$ different models (where $n$ is the sample size); which one is the model I should choose?   Is it the model which uses all the samples? This model was never calculated during the LOOCV process! Is it the model which has the least error?
15204
I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have multiple labels?
19229
TL:DR: Is it ever a good idea to train an ML model on all the data available before shipping it to production? Put another way, is it ever ok to train on all data available and not check if the model overfits, or get a final read of the expected performance of the model?    Say I have a family of models parametrized by $\alpha$. I can do a search (e.g. a grid search) on $\alpha$ by, for example, running k-fold cross-validation for each candidate.   The point of using cross-validation for choosing $\alpha$ is that I can check if a learned model $\beta_i$ for that particular $\alpha_i$ had e.g. overfit, by testing it on the "unseen data" in each CV iteration (a validation set). After iterating through all $\alpha_i$'s, I could then choose a model $\beta_{\alpha^*}$ learned for the parameters $\alpha^*$ that seemed to do best on the grid search, e.g. on average across all folds.  Now, say that after model selection I would like to use all the the data that I have available in an attempt to ship the best possible model in production.  For this, I could use the parameters $\alpha^*$ that I chose via grid search with cross-validation, and then, after training the model on the full ($F$) dataset, I would a get a single new learned model $\beta^{F}_{\alpha^*}$   The problem is that, if I use my entire dataset for training,  I can't reliably check if this new learned model $\beta^{F}_{\alpha^*}$ overfits or how it may perform on unseen data. So is this at all good practice? What is a good way to think about this problem?
30022
I am trying to learn various cross validation methods, primarily with intention to apply to supervised multivariate analysis techniques. Two I have come across are K-fold and Monte Carlo cross-validation techniques. I have read that K-fold is a variation on Monte Carlo but I'm not sure I fully understand what makes up the definition of Monte Carlo. Could someone please explain the distinction between these two methods?
85814
When training a neural network, what difference does it make to set:   batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$   where $ ab = cd $?  To put it otherwise, assuming that we train the neural network with the same amount of training examples, how to set the optimal batch size and number of iterations? (where batch size * number of iterations = number of training examples shown to the neural network, with the same training example being potentially shown several times)  I am aware that the higher the batch size, the more memory space one needs, and it often makes computations faster. But in terms of performance of the trained network, what difference does it make?
99247
Is there an existing methodology for applying a supervised learning model to a uncertain dataset? For example, say we have a dataset with classes A and B:  +----------+----------+-------+-----------+ | FeatureA | FeatureB | Label | Certainty | +----------+----------+-------+-----------+ |        2 |        3 | A     | 50%       | |        3 |        1 | B     | 80%       | |        1 |        1 | A     | 100%      | +----------+----------+-------+-----------+   How could we train a machine learning model on this? Thanks.
25786
I just started to study about stats and models stuff. Currently, my understanding is that we use MLE to estimate the best parameter(s) for a model. However, when I try to understand how the neural networks work, it seems like they commonly use another approach to estimate the parameters instead. Why don't we use MLE or is it possible to use MLE at all?
62060
I'm quite new to neural networks and currently I'm trying to train a non convolutional neural network on the MNIST data set. I'm observing some behaviour I don't quite understand.   This is the code written with keras as the library:  # the data, split between train and test sets (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()  y_train = keras.utils.to_categorical(y_train, 10) y_test = keras.utils.to_categorical(y_test, 10) model = Sequential()  early_stopper = EarlyStopping(patience=3)  model.add(Flatten(input_shape=(28,28))) model.add(Dense(128, activation="relu")) model.add(Dense(128, activation="relu")) model.add(Dense(10, activation="softmax")) model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])  model.fit(x_train, y_train, epochs=20, validation_data=(x_test, y_test), callbacks=[early_stopper])   This gives me a validation_acc of around 20%. The funny thing is, when I change the activation functions to "sigmoid", the loss function to "mean_squared_error" and the optimizer to "sgd" my performance improves to around 85% after 50 epochs.   Having read  I wonder what's the reason for the bad performance of the network I presented in the code. ReLU, cross-entropy and a dynamic optimizer like Adam all seem to improve on the idea of a very vanilla neural network with stochastic gradient optimization, mean squared error as loss and sigmoid activation functions. Yet I get a really bad performance and if I increase the number of nodes in the hidden layers I often get a network that doesn't learn at all.   EDIT: I figured out it has something to do with me not normalizing the input to values between 0 and 1 ... but why is this the problem?
43858
I am building a binary classification model for imbalanced data (e.g., 90% Pos class vs 10% Neg Class).  I already balanced my training dataset to reflect a a 50/50 class split, while my holdout (training dataset) was kept similar to the original data distribution (i.e., 90% vs 10%).  My question is regarding the validation data used during the CV hyperparameter process. During each iteration fold should:  1) Both the training and test folds be balanced  or  2) The training fold should be kept balanced while the validation fold should be made imbalanced to reflect the original data distribution and holdout dataset.  I am currently using the 1st option to tune my model; however, is this approach valid given that the holdout and validation datasets have different distributions?
122638
I'm trying to compare two classifiers using cross-validation. For one of the classifiers I'd like to use a validation set to tune a parameter. However, for the other one, I just want to use the training set and no validation set. Writing my own cross-validation loop, I'm not sure how to implement this. Is it correct to use the train set to estimate one of the classifiers and only 80% of the train set to estimate the other? This way, I could then tune one of the models and use the same test set eventually.
117577
I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc)  In many cases hyperparameter tuning on the whole dataset using either Grid or Random search is too costly time-wise.  I started using the following technique   Sub sample my dataset Use the obtained fraction to tune the hyperparameters on Use the obtained parameters to train a model using the whole dataset   To evaluate each set of parameters on the second step I use sklearn's GridSearchCV with cv=10. To evaluate the final model that I create in the third step I use sklearn's cross_val_predict. In that sense I evaluate my models leaving a 10% percent of data out, I train on the rest and measure the predictive accuracy on the 10%, iteratively 10 times, then taking the average of the scores.  What made me worry is that the prediction accuracy I get from training on my whole dataset, is really close to the evaluation I get when tuning the parameters for the best set of parameters (each tested set of parameters outputs a score obtained from averaging 10-fold-cross validation results).  Most of the times the accuracy that cross_val_predict measured using all the training examples (whole dataset) is a little bit above what the evaluation of the best parameters returned.  To illustrate this here is the evaluation of a set of parameters (on a smaller dataset than what I described above but the effect is the same)  Best parameters set found on development set: {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} Scores for all sets of parameters 0.851 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.5} 0.852 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.1} 0.829 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.001} 0.853 (+/-0.006) for {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} ...   And here are the averaged scores (from cross_val_predict) I got from training on my whole dataset using the best parameters  precision    recall  f1-score   support        0       0.86      0.85      0.86     15417       1       0.86      0.87      0.87     16561  avg / total       0.86      0.86      0.86     31978  acc score: 0.863750078179 roc au score: 0.863370490059 [[13147  2270]  [ 2087 14474]]   As you can see training on the whole dataset improves the results. I have also validated that badly tuned model (e.g. using the default values or random values for C and gamma) leads to much worse prediction accuracy.   Overall I think that tuning the hyperparameters on a subset is not ideal but can potentially lead to relatively good results without having to wait too long. I for example before using that approach used optunity package for tuning the hyperparameter on the whole dataset. This procedure would take 3-5 days to complete and would produce results that either had really good precision or really good recall but not both, so although for each class either the precision or the recall was really high (higher than what any of my other classifiers had achieved) the f1 meassure was really low. In the contrary using the later approach leads to some hours of training and a better f1 meassure.   My concerns are:  Do I limit my classification accuracy? Do I avoid using all the prediction power that my dataset can offer by tuning only on a subset? If such a harm of performance is happening is it somehow limited by some factor?
65147
I have a question regarding annotating text data for classification.  Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.   Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach). Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).   Thoughts,   will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves. will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)
87872
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.  My question is, are there any rules of thumb that tell us when we should subsample the large category in order to force some kind of balancing in the dataset.  Examples:   If the number of positive examples is 1,000 and the number of negative examples is 10,000, should I go for training my classifier on the full dataset or I should subsample the negative examples? The same question for 1,000 positive example and 100,000 negative. The same question for 10,000 positive and 1,000 negative. etc...
45382
I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).  Why is this a good idea?