I'm currently using cross_val_score and KFold to assess the impact of using StandardScaler at different points within data pre-processing, specifically whether scaling the entire training dataset prior to performing cross validation introduces data leakage and what the effect of this is when compared to scaling the data from within a Pipeline (and therefore only applying it to the training folds).  my current process is as follows:  Experiment A   Import the boston housing dataset from sklearn.datasets and split into Data (X) and target (y) create a Pipeline (sklearn.pipeline), that applies StandardScaler before applying linear regression Specify the cross validation method as KFold with 5 folds Perform cross validation (cross_val_score) using the above Pipeline and KFold method and observe the score   Experiment B   Use the same boston housing data as above fit_transform StandardScaler on the entire dataset Use cross_val_Score to perform cross validation on again 5 folds but this time input LinearRegression directly rather than a pipeline Compare the scores here to Experiment A   The scores obtained are identical (to around 13 decimal places) which I question as surely Experiment B introduces Data Leakage during cross validation.  I've seen posts stating that it doesnt matter whether scaling is done on the entire training set before cross validation, if this is true I'm looking to understand why, if this isn't true I'd like to understand why the scores can still be so similar despite the data leakage?  See my test code below:  import numpy as np import pandas as pd from sklearn.pipeline import Pipeline from sklearn import datasets  from sklearn.preprocessing import StandardScaler from sklearn.svm import LinearSVC from sklearn.model_selection import KFold, StratifiedKFold  from sklearn.model_selection import cross_val_score, cross_val_predict  from sklearn.linear_model import LinearRegression  np.set_printoptions(15)  boston = datasets.load_boston() X = boston["data"] y = boston["target"]   scalar = StandardScaler() clf = LinearRegression()  class StScaler(StandardScaler):     def fit_transform(self,X,y=None):         print('Length of Data on which scaler is fit on =', len(X))         output = super().fit(X,y) #         print('mean of scalar =',output.mean_)         output = super().transform(X)         return output   pipeline = Pipeline([('sc', StScaler()), ('estimator', clf)])  cv = KFold(n_splits=5, random_state=42)  cross_val_score(pipeline, X, y, cv = cv)    # Now fitting Scaler on whole train data   scaler_2 = StandardScaler() clf_2 = LinearRegression()  X_ss = scaler_2.fit_transform(X) cross_val_score(clf_2, X_ss, y, cv=cv)   Thanks!
123279
The  for cross-validation says the following about using feature-scaling and cross-validation:      Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction   I understand the reason behind this is to prevent information leakage between the training &amp; test sets during cross-validation, which could result in an optimistic estimate of model performance.  I am wondering then, if I wish to use Principal Component Analysis to reduce the size of a feature set before training say a regression model, and PCA requires feature-scaling to be effective, how do I chain feature-scaling to PCA to cross-validated regression, without introducing data leakage between the train-test splits in the cross-validation?
103046
I'm unclear on the exact process of using the validation data.  Let's say that I fit my neural network model and adjust hyperparameters using the training set and validation set. Do I then evaluate the test set on this model? Or do I recombine the validation and training sets and fit a fresh model with the hyperparameters that I found during the validation phase, and then evaluate on the test data? I have seen a number of different notebooks and examples that do both ways.  Surely, once I've found my hyperparameters, it makes sense to fit a fresh model using the full training set (recombined with validation set), since the validation loss has no effect on the weights.
19229
TL:DR: Is it ever a good idea to train an ML model on all the data available before shipping it to production? Put another way, is it ever ok to train on all data available and not check if the model overfits, or get a final read of the expected performance of the model?    Say I have a family of models parametrized by $\alpha$. I can do a search (e.g. a grid search) on $\alpha$ by, for example, running k-fold cross-validation for each candidate.   The point of using cross-validation for choosing $\alpha$ is that I can check if a learned model $\beta_i$ for that particular $\alpha_i$ had e.g. overfit, by testing it on the "unseen data" in each CV iteration (a validation set). After iterating through all $\alpha_i$'s, I could then choose a model $\beta_{\alpha^*}$ learned for the parameters $\alpha^*$ that seemed to do best on the grid search, e.g. on average across all folds.  Now, say that after model selection I would like to use all the the data that I have available in an attempt to ship the best possible model in production.  For this, I could use the parameters $\alpha^*$ that I chose via grid search with cross-validation, and then, after training the model on the full ($F$) dataset, I would a get a single new learned model $\beta^{F}_{\alpha^*}$   The problem is that, if I use my entire dataset for training,  I can't reliably check if this new learned model $\beta^{F}_{\alpha^*}$ overfits or how it may perform on unseen data. So is this at all good practice? What is a good way to think about this problem?
83092
I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this  that explains the correct way to cross validate when oversampling data using SMOTE technique. I have created a model using AdaBoost algorithm and set the following parametres to be used in Grid Search:     ada = AdaBoostClassifier(n_estimators=100, random_state=42)     params = {         'n_estimators': [50, 100, 200],         'random_state': [42]     }  According to the article, this is the wrong way to oversample:     X_train_upsample, y_train_upsample = SMOTE(random_state=42).fit_sample(X_train, y_train)          # cross-validate using grid search          grid_naive_up = GridSearchCV(ada, param_grid=params, cv=kf,                                   scoring='recall').fit(X_train_upsample,                                                         y_train_upsample)     grid_naive_up.best_score_  0.6715940782827282     # test set     recall_score(y_test, grid_naive_up.predict(X_test))  0.2824858757062147 Whereas the correct way to oversample is like so:     from imblearn.pipeline import Pipeline, make_pipeline          imba_pipeline = make_pipeline(SMOTE(random_state=42),                                    AdaBoostClassifier(n_estimators=100, random_state=42))     cross_val_score(imba_pipeline, X_train, y_train, scoring='recall', cv=kf)     new_params = {'adaboostclassifier__' + key: params[key] for key in params}     grid_imba = GridSearchCV(imba_pipeline, param_grid=new_params, cv=kf, scoring='recall',                             return_train_score=True)     grid_imba.fit(X_train, y_train);       # How well do we do on our validation set?     grid_imba.best_score_  0.29015614186873506     # compare this to the test set:     y_test_predict = grid_imba.predict(X_test)  0.2824858757062147 So, according to the article, the first method is wrong because when upsampling before cross validation, the validation recall isn't a good measure of the test recall (28.2%). However when using the imblearn pipeline for upsampling as part of the cross validation, the validation set recall (29%) was a good estimate of the test set recall (28.3%). According to the article, the reason for this is:  When upsampling before cross validation, you will be picking the most oversampled model, because the oversampling is allowing data to leak from the validation folds into the training folds.  Can anyone explain to me simply how the oversampling allows data to leak into the validation and causes the overfitting? And why does this problem not occur in the imblearn pipeline?
61395
Computing power considerations aside, are there any reasons to believe that increasing the number of folds in cross-validation leads to better model selection/validation (i.e. that the higher the number of folds the better)?  Taking the argument to the extreme, does leave-one-out cross-validation  necessarily lead to better models than $K$-fold cross-validation?  Some background on this question: I am working on a problem with very few instances (e.g. 10 positives and 10 negatives), and am afraid that my models may not generalize well/would overfit with so little data.
104639
I searched on topics of the bias and variance trade-off and got back lots of questions with different levels of response. The information is scattering too much and unsystematic to answer my own question which stuck me. So I open a new question here and would like to hackle how I generally run a machine learning modelling process and ask for help on the issue of bias-variance assessment where I was confused or had no idea. If anyone noticed any useful posts to my questions please help point out the links and many thanks.  I execute the process basically according to the instruction below:    Question 1: When I start at step one, I try to think an issue that all my following analysis will be based on this train-test split. How about another split and is it possible to cause an obvious change in analysis results (final models, performance metrics, important features etc.)? If so, that means the analysis was split-dependent.   I have some thoughts here: assuming there exists an unknown underlying distribution of the domain set X (the whole dataset except the target class y), if the sample distribution of training set S was approximately similar to that of test set T, then the learning process should reach a good performance. Otherwise, if the two sample distributions differed in some non-trivial way, then the learner/classifier from training set S may not probably do a good job on test set T.     I made a trial to execute a 10-fold cross validation on the whole dataset and to see how variable the ROC AUC score was. In my particular dataset, I got a set of 10 scores with large variation (from maximum 0.85 to minimum 0.59). This led to my worry whether the current split I adopted for analysis was poor one.  To expand Question 1, I would like to ask further: what is the topic of this issue in machine learning theory? Is it the model stability or model variance?  I tuned the hyper-parameter/parameters in step 2 via cross validation. I try to use validation curve (refer to ) as below to choose the parameter value and avoid under/over-fitting problems.     Question 2: If this curve worked well in detecting under/over-fitting problems, what metric should be used: accuracy score/error rate, presicion, recall, f1 or roc_auc? (My current project was a classification problem with an imbalanced binary target class.)  According to the definition of bias, average error rate could be the measure of bias, so accuracy score/error rate seemed more reasonable in bias/variance issue. In fact, I concerned roc_auc score more than accuracy itself.   Question 3: how can we measure the variance? I saw a plot  and the gap was said to be able to indicate the potential variance (poorer performance on validation set than that on training set). Any other (better) way/metric can be used for this point?    In my analysis, the bias/variance check was run in the parameter tuning stage which seemed to return me a candidate model with a balanced bias/variance combination.  Question 4: does my bias/variance analysis miss some steps and how do I adjust it?  many ensemble learning methods have their own way to reduce variance or bias and  I hope to objectively evaluate the bias/variance for each candidate model.   Thanks for reading my long question but it really help for me to deliver a professional practice in my daily work. Many thanks again.
32415
Usually regression models are evaluated using $R^2$. I understand this metric can be misleading too at times but as far as I understand the first parameter we look at is $R^2$.   There is another parameter which is often used and it is $MAPE$.  Both are functions of errors between predicted and true value. I am just wondering if there are certain cases where one should be preferred above another?   $R^2$ can become negative if being used on test data (on which the model is not built). Is this a reason to not use $R^2$ in these cases?  Can anyone give a qualitative insight on where $MAPE$ should be used and where $R^2$ should be used.
104323
/edit: Further follow up now you can use     /edit: following up on my own post.  irlba now has "center" and "scale" arguments, which let you use it to calculate principle components, e.g:  pc &lt;- M %*% irlba(M, nv=5, nu=0, center=colMeans(M), right_only=TRUE)$v    I have a large, sparse Matrix of features I would like to use in a machine learning algorithm:  library(Matrix) set.seed(42) rows &lt;- 500000 cols &lt;- 10000 i &lt;- unlist(lapply(1:rows, function(i) rep(i, sample(1:5,1)))) j &lt;- sample(1:cols, length(i), replace=TRUE) M &lt;- sparseMatrix(i, j)   Because this matrix has many columns, I would like to reduce its dimensionality to something more manageable.  I can use the excellent  to perform SVD and return the first n principal components (5 shown here; I'll probably use 100 or 500 on my actual dataset):  library(irlba) pc &lt;- irlba(M, nu=5)$u   However, I've read that prior to performing PCA, one should center the matrix (subtract the column mean from each column).  This is very difficult to do on my dataset, and furthermore would destroy the sparsity of the matrix.  How "bad" is it to perform SVD on the un-scaled data, and feed it straight into a machine learning algorithm?  Are there any efficient ways I could scale this data, while preserving the sparsity of the matrix?    /edit: A brought to my attention by B_miner, the "PCs" should really be:  pc &lt;- M %*% irlba(M, nv=5, nu=0)$v    Also, I think whuber's answer should be pretty easy to implement, via the crossprod function, which is extremely fast on sparse matrices:  system.time(M_Mt &lt;- crossprod(M)) # 0.463 seconds system.time(means &lt;- colMeans(M)) #0.003 seconds   Now I'm not quite sure what to do to the means vector before subtracting from M_Mt, but will post as soon as I figure it out.    /edit3: Here's modified version of whuber's code, using sparse matrix operations for each step of the process.  If you can store the entire sparse matrix in memory, it works very quickly:  library('Matrix') library('irlba') set.seed(42) m &lt;- 500000 n &lt;- 100 i &lt;- unlist(lapply(1:m, function(i) rep(i, sample(25:50,1)))) j &lt;- sample(1:n, length(i), replace=TRUE) x &lt;- sparseMatrix(i, j, x=runif(length(i)))  n_comp &lt;- 50 system.time({   xt.x &lt;- crossprod(x)   x.means &lt;- colMeans(x)   xt.x &lt;- (xt.x - m * tcrossprod(x.means)) / (m-1)   svd.0 &lt;- irlba(xt.x, nu=0, nv=n_comp, tol=1e-10) }) #user  system elapsed  #0.148   0.030   2.923   system.time(pca &lt;- prcomp(x, center=TRUE)) #user  system elapsed  #32.178   2.702  12.322  max(abs(pca$center - x.means)) max(abs(xt.x - cov(as.matrix(x)))) max(abs(abs(svd.0$v / pca$rotation[,1:n_comp]) - 1))   If you set the number of columns to 10,000 and the number of principal components to 25, the irlba-based PCA takes about 17 minutes to calculate 50 approximate principal components and consumes about 6GB of RAM, which isn't too bad.
24206
My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect negatively algorithms because the feature with bigger values will take more influence, is it a good practice to simply ALWAYS scale/normalize the data?  It looks to me that if the values are already similar among then, then normalizing them will have little effect, but if the values are very different normalization will help, however it feels too simple to be true :)  Am I missing something? Are there situations/algorithms were actually it is desirable to let some features to deliberately outweigh others?
39206
I have dataset of 15 features and the goal is to estimate a best fitting curve (regression). Now I want to use a deep learning technique for this purpose. Now I see that there are many architectures that can be used such as CNN, DBM or auto-encoder, etc.. Which one to go for. And I could not find any examples for how to use the same. A sample code would be of great help. Thanks  P.S. Some of the features are categorical. How do we handle that as well?
4158
A common good practice in Machine Learning is to do feature normalization or data standardization of the predictor variables, that's it, center the data substracting the mean and normalize it dividing by the variance (or standard deviation too). For self containment and to my understanding we do this to achieve two main things:   Avoid extra small model weights for the purpose of numerical stability. Ensure quick convergence of optimization algorithms like e.g. Conjugate Gradient so that the large magnitude of one predictor dimension w.r.t. the others doesn't lead to slow convergence.   We usually split the data into training, validation and testing sets. In the literature we usually see that to do feature normalization they take the mean and variance (or standard deviation) over the whole set of predictor variables. The big flaw I see here is that if you do that, you are in fact introducing future information into the training predictor variables namely the future information contained in the mean and variance.  Therefore, I do feature normalization over the training data and save the mean and variance. Then I apply feature normalization to the predictor variables of the validation and test data sets using the training mean and variances. Are there any fundamental flaws with this? can anyone recommend a better alternative?
62060
I'm quite new to neural networks and currently I'm trying to train a non convolutional neural network on the MNIST data set. I'm observing some behaviour I don't quite understand.   This is the code written with keras as the library:  # the data, split between train and test sets (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()  y_train = keras.utils.to_categorical(y_train, 10) y_test = keras.utils.to_categorical(y_test, 10) model = Sequential()  early_stopper = EarlyStopping(patience=3)  model.add(Flatten(input_shape=(28,28))) model.add(Dense(128, activation="relu")) model.add(Dense(128, activation="relu")) model.add(Dense(10, activation="softmax")) model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])  model.fit(x_train, y_train, epochs=20, validation_data=(x_test, y_test), callbacks=[early_stopper])   This gives me a validation_acc of around 20%. The funny thing is, when I change the activation functions to "sigmoid", the loss function to "mean_squared_error" and the optimizer to "sgd" my performance improves to around 85% after 50 epochs.   Having read  I wonder what's the reason for the bad performance of the network I presented in the code. ReLU, cross-entropy and a dynamic optimizer like Adam all seem to improve on the idea of a very vanilla neural network with stochastic gradient optimization, mean squared error as loss and sigmoid activation functions. Yet I get a really bad performance and if I increase the number of nodes in the hidden layers I often get a network that doesn't learn at all.   EDIT: I figured out it has something to do with me not normalizing the input to values between 0 and 1 ... but why is this the problem?
118172
I am learning how to use libsvm through sklearn.svm in python. I read  about what happens and why when you change the C value as part of your model. My intuition from what I've learned would be that lower C values would use less support vectors to make a more general classification, while higher C values would use more support vectors to attempt to 'overfit' and account for all outliers.   That is not the case. For an example where I looped through a set of C values like so:  print(c) model = svm.SVC(kernel='linear', C=c) model.fit(Xtrain, ytrain) print("support vectors:", len(model.support_))   I got results:  1.0 support vectors: 1810 10.0 support vectors: 1750 100.0 support vectors: 1626 1000.0 support vectors: 1558   As you can see, as C goes up, the number of support vectors used in the model goes down. Why does this happen?
89991
I've been using the $K$-fold cross-validation a few times now to evaluate performance of some learning algorithms, but I've always been puzzled as to how I should choose the value of $K$.  I've often seen and used a value of $K = 10$, but this seems totally arbitrary to me, and I now just use $10$ by habit instead of thinking it over. To me it seems that you're getting a better granularity as you improve the value of $K$, so ideally you should make your $K$ very large, but there is also a risk to be biased.  I'd like to know what the value of $K$ should depend on, and how I should be thinking about this when I evaluate my algorithm. Does it change something if I use the stratified version of the cross-validation or not?
49384
I'm using Python Keras package for neural network. This is the . Is batch_size equals to number of test samples? From Wikipedia we have  information:     However, in other cases, evaluating the sum-gradient may require   expensive evaluations of the gradients from all summand functions.   When the training set is enormous and no simple formulas exist,   evaluating the sums of gradients becomes very expensive, because   evaluating the gradient requires evaluating all the summand functions'   gradients. To economize on the computational cost at every iteration,   stochastic gradient descent samples a subset of summand functions at   every step. This is very effective in the case of large-scale machine   learning problems.   Above information is describing test data? Is this same as batch_size in keras (Number of samples per gradient update)?
86598
When running hierarchical clustering analysis of a matrix of individuals x samples (e.g., employee performances across different days), there are several possibilities for normalization. If one is clustering the columns (to see whether on certain days individuals perform similarly), one could   z-score normalize across the rows to make each individual employees mean and standard deviation comparable across days, or z-score normalize across the columns to make all employees comparable within a day, or  not normalize at all and cluster the raw values   Could someone explain the relative advantages/disadvantages of each approach here? To clarify, I am using correlation distance.  Methods 1 or 2 in practice give different results but it's not clear that for the task of seeing if days cluster together, whether #1 or #2 are more appropriate if one chooses to normalize.
22939
I am wondering how to choose a predictive model after doing K-fold cross-validation.   This may be awkwardly phrased, so let me explain in more detail: whenever I run K-fold cross-validation, I use K subsets of the training data, and end up with K different models.   I would like to know how to pick one of the K models, so that I can present it to someone and say "this is the best model that we can produce."   Is it OK to pick any one of the K models? Or is there some kind of best practice that is involved, such as picking the model that achieves the median test error?