I am working on a data set that has 930 rows and 5 columns. I'm implementing KNN Regression with bootstrapping and my RMSE is almost 400 (depending on different cross validated outputs). I read  and one of the answers mentioned that if the training and test RMSE are similar, then the model is good; any my training error is close to 400 too. I've tried searching online for reliable sources to know if this high an error rate is acceptable, but I've mainly read articles and posts that say the closer the RMSE is to 0, the better, which definitely is not the case here.   Any advice is appreciated!
4773
I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to interpret them. In short, how do I tell if my model is any good based on the metric? Is an $R^2$ of 0.6 (for example) sufficient to let me proceed to draw inferences or base scientific/business decisions?     This question is intentionally broad, to cover a wide variety of situations that members frequently encounter; such questions could be closed as duplicates of this one. Edits to broaden the scope beyond the metrics mentioned here are welcome, as are additional answers - particularly those that offer insight about other classes of metrics.
88722
I found two questions  and  about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my Convolution Neural Network. What does that mean?
69494
Let's say I have some data, and then I fit the data with a model (a non-linear regression). Then I calculate the R-squared ($R^2$).  When R-squared is negative, what does that mean? Does that mean my model is bad? I know the range of $R^2$ can be [-1,1]. When $R^2$ is 0, what does that mean as well?
81288
I have recently been reading a lot on this site (@Aniko, @Dikran Marsupial, @Erik) and elsewhere about the problem of overfitting occuring with cross validation - (Smialowski et al 2010 Bioinformatics, Hastie, Elements of statistical learning).  The suggestion is that any supervised feature selection (using correlation with class labels) performed outside of the model performance estimation using cross validation (or other model estimating method such as bootstrapping) may result in overfitting.   This seems unintuitive to me - surely if you select a feature set and then evaluate your model using only the selected features using cross validation, then you are getting an unbiased estimate of generalized model performance on those features (this assumes the sample under study are representive of the populatation)?  With this procedure one cannot of course claim an optimal feature set but can one report the performance of the selected feature set on unseen data as valid?  I accept that selecting features based on the entire data set may resuts in some data leakage between test and train sets. But if the feature set is static after initial selection, and no other tuning is being done, surely it is valid to report the cross-validated performance metrics?   In my case I have 56 features and 259 cases and so #cases > #features. The features are derived from sensor data.  Apologies if my question seems derivative but this seems an important point to clarify.  Edit: On implementing feature selection within cross validation on the data set detailed above (thanks to the answers below), I can confirm that selecting features prior to cross-validation in this data set introduced a significant bias. This bias/overfitting was greatest when doing so for a 3-class formulation, compared to as 2-class formulation.  I think the fact that I used stepwise regression for feature selection increased this overfitting; for comparison purposes, on a different but related data set I compared a sequential forward feature selection routine performed prior to cross-validation against results I had previously obtained with feature selection within CV. The results between both methods did not differ dramatically. This may mean that stepwise regression is more prone to overfitting than sequential FS or may be a quirk of this data set.
30986
I wonder if someone knows any general rules of thumb regarding the number of bootstrap samples one should use, based on characteristics of the data (number of observations, etc.) and/or the variables included?
32233
I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max 20-30 for later use with model training (binary classification - increase / decrease).  I was thinking about using random forests for feature ranking. Is it a good idea to use them recursively? For example, let's say in the first round I drop the worst 20%, second too and so on until I get the desired number of features.  Should I use cross-validation with RF? (It's intuitive for me not to use CV because that's pretty much what RF does already.)  Also if I go with random forests should I use them as classifiers for the binary or regressor for the actual increase / decrease to get feature importances?  By the way, the models I would like to try after feature selection are: SVM, neural nets, locally weighted regressions, and random forest. I'm mainly working in Python.
95184
What is the minimum number of data points required for a kernel density estimation to be considered non-misleading/acceptable/adequate?  Is there a some rule based on how dispersed the data is?  For example, if the inter-quartile range is "small enough", do you need less points than if the inter-quartile range is "larger"?  Alternatively, you may have encountered a situation where a certain size of dataset was "enough" or "not enough" and can share the reasons why KDE worked/didn't work in that specific case?  Criteria to assess accuracy:  If you had two samplesfrom the same population and you do a KDE on each one, then the two resulting densities are very similar.
13209
I am working with many algorithms: RandomForest, DecisionTrees, NaiveBayes, SVM (kernel=linear and rbf), KNN, LDA and XGBoost. All of them were pretty fast except for SVM. That is when I got to know that it needs feature scaling to work faster. Then I started wondering if I should do the same for the other algorithms.
107010
While reading , I stumbled across the following (p. 210):     [...] the model containing all of the predictors will always have the smallest $RSS$ and the largest $R^2$ , since these quantities are related to the training error.   The text talks about feature (input) selection in regressive models. The reference is the following (p. 205):     The problem is that a low $RSS$ or a high $R^2$ indicates a model with a low training error, whereas we wish to choose a model that has a low test error.   I don't understand why the authors relate those error measures to the training dataset, since we can calculate $RSS$ or $R^2$ on a test dataset (given that I have enough data) too.  Is there any reason for this distinction?
60677
I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree works the best with training accuracy of 1.0 and test accuracy of 0.5. The neural networks, which I believed would always perform the best no matter what has a training accuracy of 0.92 and test accuracy of 0.42 which is 8% less than the decision tree classifier.   Could someone please explain the circumstances/cases where neural networks could have low accuracy when compared to a modelling technique like the decision tree. I had tried my neural network with different configurations like:  1 hidden layer and 1 neuron : Train Accuracy 34% Test Accuracy 42% 7 hidden layers and 5 neurons in each layer: Train Accuracy 79% Test Accuracy 42% 1 hidden layer and 100 neurons: Train Accuracy 34% and Test Accuracy 35%   but not in a single case, I found the neural network to beat the decision tree test accuracy of 50%.
117823
I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem is I'm not sure how to interpret the success of my final result.   For example if I achieved a RMSLE of $1.052$ could I raise it the the exponential power $e$ and interpret it like rmse? (ie. $e^{1.052}=2.863=RMSE$)?  Could I then say that my predictions were $\pm \$2.863$ on average from the the actual prices? Or is there a better way to interpret the metric? Or can the metric even be interpreted at all with the exception of comparing to the other RMSLEs of other models?
52356
Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of bootstrap samples with replacement, and calculate my statistic $T$ on each bootstrap sample to create an estimated distribution for $T$.  What are examples where this is a bad idea?  For example, one case where naively performing this bootstrap would fail is if I'm trying to use the bootstrap on time series data (say, to test whether I have significant autocorrelation). The naive bootstrap described above (generating the $i$th datapoint of the nth bootstrap sample series by sampling with replacement from my original series) would (I think) be ill-advised, since it ignores the structure in my original time series, and so we get fancier bootstrap techniques like the block bootstrap.  To put it another way, what is there to the bootstrap besides "sampling with replacement"?
113258
I got ValueError when predicting test data using a RandomForest model.  My code:  clf = RandomForestClassifier(n_estimators=10, max_depth=6, n_jobs=1, verbose=2) clf.fit(X_fit, y_fit)  df_test.fillna(df_test.mean()) X_test = df_test.values   y_pred = clf.predict(X_test)   The error:  ValueError: Input contains NaN, infinity or a value too large for dtype('float32').   How do I find the bad values in the test dataset? Also, I do not want to drop these records, can I just replace them with the mean or median?  Thanks.
104288
I'm learning Neural Networks and I grasped the algebra behind them. I'm now interested in understanding how normalization and centering of the input data affect them. In my personal learning project (Regression with NN) I've transformed my input variables to a range between 0 and 1 using the following function:  normalize &lt;- function(x) {return((x - min(x))/ (max(x) - min(x)))}   The NN model fits well and has an acceptable out-of-sample prediction error.   However, I read in other  that scaling the inputs to have mean 0 and a variance of 1 is advised for NN. I don't fully understand:   how this transformation works better for NN against the min-max normalization between 0 and 1. how can I assess which transformation to apply in my data?
52264
I have two models that predicted a value that is area based. I want to measure the RMSE to compare the models, but I would like it to be weighted for area so that errors on a large area are given more weight then errors on a small area. I have come up with the following function in R:  weighted.rmse &lt;- function(actual, predicted, weight){   sqrt(mean((predicted-actual)^2*weight/sum(weight))) }   For weight I will use the area of prediction. So to my question, is this a valid method for model comparison? Is there a better way?
85834
Appreciate your wisdom on this,  My understanding is most of the tutorials recommend normalizing / scaling the data prior feeding the tensorflow models. Doesn't normalization require that data conforms to the normal parametric distribution?  What good is a non-linear model if scaling / normalizing is a pre-requisite prior using the non-linear model such as tensorflow?   So back to the question, should I always normalize / scale my data prior feeding my tensorflow models?  thanks
121542
I trained a model with some algorithms like random forest, logistic regression and so on.   My dataset was split into 80% CV train data (so actually 60% of data to train the model and 20 % for testing with cv). I built my model and now I'm using the last 20% (which I didn't used the whole time) but I'm wondering why is the predictive accuracy better with the validation set than with the training set?    specificity validation: 0.62962963 sensitivity validation: 0.94761905   on the left is the mean, on the right is the standard deviation (+/-) with a 10 times cross validation    specificity_train: 0.55  (+/- 0.10)  recall_train: 0.94   (+/- 0.02)   Did I something wrong? Thought this is the normal / best way to test, if the model is overfitted or not.  Or is it OK due to the standard deviation?