I want someone to answer me if what I'm doing is correct. I have a labelled data that I want to train different machine learning models that can predict the outcome. here are the steps that I have went through:   I have divided the data into two sets 80% for training and 20% for testing. I have cross validated the training set (and only the training set) with 10 folds using different models (Knn, ANN, SVM ,...etc).  I kept tuning the parameters of the models until I got a satisfactory root mean squared error (RMSE) for each model. I used the parameters that produced the lowest RMSE to build each model using the training set (80% of the data). I fed the testing set (the remaining 20%) into the each model and got a prediction from each model. I evaluated the testing set prediction error of each model using MSE, RMSE,MAPE and MAE. Compare the models and recommend the model that produced the lowest error.   My questions:   is using 10-fold cross validation on the testing set alone is similiar to dividing the data into 70% training, 10 % validation and 20% testing? It would be really helpful if you could provide me with research papers that adopt such technique.  Does this procedure makes sense, or am I doing something wrong?
122638
I'm trying to compare two classifiers using cross-validation. For one of the classifiers I'd like to use a validation set to tune a parameter. However, for the other one, I just want to use the training set and no validation set. Writing my own cross-validation loop, I'm not sure how to implement this. Is it correct to use the train set to estimate one of the classifiers and only 80% of the train set to estimate the other? This way, I could then tune one of the models and use the same test set eventually.
117702
I am new to machine learning and would like to know if it makes sense to fix the number of estimators and the maximal depth of a random forest with cross validation ?  My intuition would be that yes, as cross validation enables one to determine models' hyper parameters. Nevertheless, I think I heard a professor saying that it is illogical to estimate random forests hyperparameters with such procedures... Maybe because the number of estimators would be chosen based on the training data, and then there would be a kind of overfitting as the training and testing datasets are not of the same size, and for instance, having a bigger set size might permit one to consider deeper trees, or maybe averaging on more trees would be necessary to reduce the variance...  What are the other possible ways to determine Random Forests hyper-parameters to use ?  More generally, what are the main model selection procedures in machine learning ?
72778
I've been using the  in R to build predictive models for classification and regression.  Caret provides a unified interface to tune model hyper-parameters by cross validation or boot strapping.  For example, if you are building a simple 'nearest neighbors' model for classification, how many neighbors should you use?  2? 10? 100? Caret helps you answer this question by re-sampling your data, trying different parameters, and then aggregating the results to decide which yield the best predictive accuracy.  I like this approach because it is provides a robust methodology for choosing model hyper-parameters, and once you've chosen the final hyper-parameters it provides a cross-validated estimate of how 'good' the model is, using accuracy for classification models and RMSE for regression models.  I now have some time-series data that I want to build a regression model for, probably using a random forest. What is a good technique to assess the predictive accuracy of my model, given the nature of the data? If random forests don't really apply to time series data, what's the best way to build an accurate ensemble model for time series analysis?
23122
I am familiar with "regular" cross-validation, but now I want to make timeseries predictions while using cross-validation with a simple linear regression function. I write down a simple example, to help clarify my two questions: one about the train/test split, one question about how to train/test for models when the aim is to predict for different n, with n the steps of prediction, in advance.  (1) The data  Assume I have data for timepoints 1,...,10 as follows:  timeseries = [0.5,0.3,10,4,5,6,1,0.4,0.1,0.9]   (2) Transforming the data into a format useful for supervised learning  As far as I understand, we can use "lags", i.e. shifts in the data to create a dataset suited for supervised learning:  input = [NaN,0.5,0.3,10,4,5,6,1,0.4,0.1] output/response = [0.5,0.3,10,4,5,6,1,0.4,0.1,0.9]   Here I have simply shifted the timeseries by one for creating the output vector. As far as I understand, I could now use input as the input for a linear regression model, and output for the response (the NaN could be approximated our replaced with a random value).  (3) Question 1: Cross-validation ("backtesting")  Say I want to do now 2-splits, do I have to shift the train as well as the test sets?    I.e. something like:  Train-set:  Independent variable: [NaN,0.5,0.3,10,4,5]  Output/response variable:[0.5,0.3,10,4,5,6]  Test-set:  Independent variable: [1,0.4,0.1]  Output/response variable:[0.4,0.1,0.9]  (ii) Question 2: Predicting different lags in advance:  As obvious, I have shifted dependent to independent variables by 1. Assuming now I would like to train a model which can predict 5 time steps in advance -- can I keep this lag of one, and nevertheless use the model to predict n+1,...,n+5,... or do I change the shift from independent to dependent variable to 5? What exactly is the difference?
93962
Is it possible to have test error lower than training error?  I have a classification problem with 2000 samples, 500 of which are positives, 1500 are negatives. I split my data into 70% training data, 30% test data.  Run random forest with 200 estimators and cv=10. I did this several times and compared the recall and precision score and notice the scores for my test set is significantly better. Is this possible?
35203
I'm having trouble understanding the ROC curve.  Is there any advantage / improvement in area under the ROC curve if I build different models from each unique subset of the training set and use it to produce a probability? For example, if $y$ has values of $\{a, a, a, a, b, b, b, b\}$, and I build model $A$ by using $a$ from 1st-4th values of $y$ and 8th-9th values of $y$ and build model $B$ by using remained train data. Finally, generate probability. Any thoughts / comments will be much appreciated.   Here is r code for better explanation for my question:  Y    = factor(0,0,0,0,1,1,1,1) X    = matirx(rnorm(16,8,2)) ind  = c(1,4,8,9) ind2 = -ind  mod_A    = rpart(Y[ind]~X[ind,]) mod_B    = rpart(Y[-ind]~X[-ind,]) mod_full = rpart(Y~X)  pred = numeric(8) pred_combine[ind]  = predict(mod_A,type='prob') pred_combine[-ind] = predict(mod_B,type='prob') pred_full          = predict(mod_full, type='prob')   So my question is, area under ROC curve of pred_combine vs pred_full.
41967
In R, I'm running a linear model of the form:  fit &lt;- lm(cbind(Y1,Y2,Y2) ~ X1 + X2 + X3, data=regression_table);   Then I'm using predict to generate predictions;  predicted_vals &lt;- predict(fit, test_data);   Does any one have suggestions on how I should go about evaluating the training error of this multivariate model? I understand the notion of cross validation for univariate multiple models, though I'm not sure how to go about go about doing something similar for a multivariate model.
72427
According to  (also  and ), we cannot deduce just from the data alone (without any domain knowledge) which classifier is better. Of course, we use cross-validation to do precisely that.  So what are the assumptions that we implicitly make that make cross-validation a meaningful procedure?
117823
I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem is I'm not sure how to interpret the success of my final result.   For example if I achieved a RMSLE of $1.052$ could I raise it the the exponential power $e$ and interpret it like rmse? (ie. $e^{1.052}=2.863=RMSE$)?  Could I then say that my predictions were $\pm \$2.863$ on average from the the actual prices? Or is there a better way to interpret the metric? Or can the metric even be interpreted at all with the exception of comparing to the other RMSLEs of other models?
42381
I have a deep neural network model and I need to train it on my dataset which consists of about 100,000 examples, my validation data contains about 1000 examples. Because it takes time to train each example (around 0.5s for each example) and in order to avoid overfitting, I would like to apply early stopping to prevent unnecessary computation. But I am not sure how to properly train my neural network with early stopping, several things I do not quite understand now:   What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1) Is it the case that the first few epochs might yield worse result before it starts converging to better value? In that case, should we train our network for several epochs before checking for early stopping? How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?   Thank you in advance.
74366
When we do k-fold cross validation, should we just use the classifier that has the highest test accuracy? What is generally the best approach in getting a classifier from cross validation?
44407
I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a higher test accuracy mean. In best practice, should I use paired t-test, unpaired t-test or other kinds of test? Should I use Welch's test if people are concerned about unequal variances?
22528
Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in ［0,1］$. Assume  one use gbdt to learn this linear model. And if the test data consists of ($X,Y$) such that $X \in ［4,5］$. Can the learned gbdt model correctly predict data in test set? The reason is that any node in any tree in gbdt has the form: $x_i &gt; a $ and $x_i &lt;=a$, $a$ is the splitting point of variable $x_i$, and the $X$ in test data has disjoint range with the one in train set, so the gbdt model will not be able to distinguish the $X\in ［0,1］$ in train set and $X \in ［4,5］$ in test set, more specifically, the $X\in ［4,5］$ will lie to the right side of the splitting point and then will predict the same value as the $X$ in the upper part of $［0,1］$, for example, in $［2/3,1］$. As the true model is linear, which means it wil not predict the same value for $X$ in train and test set, resulting that the gbdt will not correctly learn such a model. Is the above reasoning correct?
68491
I am running k-fold cross validation on my training data, and then choosing the best set of hyper parameters, re-training on the training data and testing on a new (unseen) testing data. I am getting the following testing results (results on the unseen data):  R^2: 1.000 Adj R^2: 1.000 RMSE: 0.081 MSE: 0.007 MAE: 0.045 MAPE: 0.058   Can $R^2$ have a value of 1.00 but RMSE have a  value of 0.081? Doesn't an $R^2$ of value 1 indicate no errors at all?
12986
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, e.g. a product.      My question is: Is it always for the best to do so? What if the   performance actually deteriorates?   For example, let us assume a case where the model scores around 65% in classifying the testing subset. This could mean that either the model is trained insufficiently OR that the testing subset consists of outliers. In the latter case, training the final model with them would decrease its performance and you find out only after deploying it.   Re-phrasing my initial question:     If you had a one-time demonstration of a model, such as deploying it   on embedded electronics on-board an expensive rocket experiment, would you trust a   model that has been re-trained with the test subset in the final step without being   re-tested on its new performance?
99785
I'm fairly new to Machine Learning/Modelling and I'd like some background to this problem. I have a dataset where the number of observations is $n&lt;200$ however the number of variables is $p\sim 8000$. Firstly does it even make sense to consider building a model on a dataset like this or should one consider a variable selection technique to start with such as ridge regression or Lasso? I've read that this situation can lead to over-fitting. Is that the case for all ML techniques or do some techniques handle this better than others? Without too much maths a simple explanation on why the maths start to breakdown for $p&gt;n$ would be appreciated.
18820
In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and then we train the model using the best hyperparameters with the training set and apply the trained model to the test to get the performance.  If we have a small dataset, we cannot create training and test set (not enough samples). Therefore, we will do cross validation (k-fold, leave-one-out, etc) to evaluate the model performance.  I have seen nested cross validation (whether repeated or stratified) has been used in the setting of small dataset, i.e., to generate generalized model performance while optimizing parameters selection. My question is, how can I obtain the best hyperparameters in nested cross validation (repeated/no repeated)?  I am interested in doing this in scikit-learn, if possible. I am a bit confused about how to do it.  I have read several resources but none gave me the definitive answer for this question: