What is the point of oversampling an imbalanced data set if the ratio of the classes needs to be preserved in Cross validation ? If I have 1000 rows in a data set where 800 rows belong to one class and 200 rows to another class, this would be considered an Imbalanced data set due to it's skewed distribution. If I over sample the smaller class by just duplicating the rows and make it 400 rows, the ratio of the major and minor class is changed. Now having done that, why would I ensure in my cross-validation folds (if I should) that the ratio of majority to minority class is still 80:20 ? What am I missing here ? Additionally, my test set might not have the same ratio of classes. How can I ensure that the skewed distribution does not impact my model ?
7818
We already had multiple questions about unbalanced data when using , , ,  and a number of other similar questions, what makes it a very popular topic! Unfortunately, each of the questions seems to be algorithm-specific and I didn't find any general guidelines for dealing with unbalanced data.  Quoting , dealing with unbalanced data     (...) heavily depends on the learning method. Most general purpose   approaches have one (or several) ways to deal with this.   But when exactly should we worry about unbalanced data? Which algorithms are mostly affected by it and which are able to deal with it? Which algorithms would need us to balance the data? I am aware that discussing each of the algorithms would be impossible on Q&amp;A site like this, I am rather looking for a general guidelines on when it could be a problem.
47279
When dealing with the class imbalance problem in a binary classifier, there are three ways I know of to address it: over-sampling, under-sampling and using cost-sensitive methods.   Are there any guidelines, rules of thumb or general strategies to choose among these methods? A possible answer would be: over-sample when the positive class has more than 100 instances (I just made it up).
22361
Assume that we have a training data set (with both features and labels) and a test data set (with only features).   When we build a machine learning model that requires normalization of the features, the correct way of doing normalization (to prevent information leakage) is to only use the training data set. Namely, a wrong way of doing normalization is to stack the training (exclude the Y column) and test data set together, and perform normalization (i.e., using the mean and variance of the entire training + test data set). The intuition here is very clear: if you want to get an unbiased estimator of your model performance in production, then when you train your model, you shouldn't instill any information in the test data set that will be used to gauge the actual performance of your model.  My question is the following: When we have trained the model correctly and about to use this model to do prediction, how should we normalize the test data set? I believe the correct way to normalize the test data set is: using the mean and variance obtained from the training data set to do normalization on the test data set.  However, why not just normalize the test data using its own mean and variance? Or why not stack train and test data set together and using the overall mean and variance to normalize the test data set? In the prediction stage, the idea and intuition of data snooping and information leakage is not clear to me.
81664
I'm aware that there are many ways of dealing with datasets where there is a strong class imbalance in the target variable: downsampling the more prevalent and less important class, over-weighting the valuable minority class, etc. Are there any loss functions that are ideally suited for these types of problems? I've heard of AUC, but AUC sounds like a means of evaluating a model once its training is complete (correct me if I'm wrong). Log loss and squared loss penalize a model's over confidence, but overconfidence and class imbalance seem unrelated. Is that true?
21856
Why does a cross-validation procedure overcome the problem of overfitting a model?
19398
I have two groups of independent samples whose means I want to compare. The data comes from a distinctly non-normal distribution. (Unimodal, but heavily right skewed).    Let's say that one group (Group A) has 14,000 observations and that another group (Group B) has 700 observations. Thus each group has a decent amount of observations (we are not talking about 2000 versus 5 here).   I am of the opinion that this strong unbalance in the group sizes will not make the Mann-Whitney-Wilcoxon Test any less applicable, however a colleague is suggesting that we randomly sample 700 observations from Group A so that our group sizes are perfectly balanced. He feels that the unbalance of the groups is making the test fail to find the "correct" result. (Where correct result means "result he wants," i.e. that the two groups are statistically significantly different).   As I said, I think that having highly unequal groups does not cause the Mann-Whitney-Wilcoxon Test to somehow stop working.   As far as I understand it, I do believe that the size of the smaller group will have implications for the power of our test, but even there, I am thinking that we probably have at least a decently sufficient sample size for Group B.   Will this highly unbalanced data somehow invalidate my analysis?
70159
I have a dataset with lots Y=0 and few Y=1. I have to run logistic regression, so I'm using a retrospective sample in order to get a more balanced sample. Could someone give me some references that explain which are the problems arising when I use logistic regression in an unbalanced sample? I kwow that the main problems are instability of estimated coefficients and poor predictive power of the model, but I need some references.
73299
When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset.  This is because the models usually have three "levels" of parameters: the first "parameter" is the model class (e.g. SVM, neural network, random forest), the second set of parameters are the "regularization" parameters or "hyperparameters" (e.g. lasso penalty coefficient, choice of kernel, neural network structure) and the third set are what are usually considered the "parameters" (e.g. coefficients for the covariates.)  Given a model class and a choice of hyperparameters, one selects the parameters by choosing the parameters which minimize error on the training set.  Given a model class, one tunes the hyperparameters by minimizing error on the validation set.  One selects the model class by performance on the test set.  But why not more partitions?  Often one can split the hyperparameters into two groups, and use a "validation 1" to fit the first and "validation 2" to fit the second.  Or one could even treat the size of the training data/validation data split as a hyperparameter to be tuned.  Is this already a common practice in some applications?  Is there any theoretical work on the optimal partitioning of data?
18820
In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and then we train the model using the best hyperparameters with the training set and apply the trained model to the test to get the performance.  If we have a small dataset, we cannot create training and test set (not enough samples). Therefore, we will do cross validation (k-fold, leave-one-out, etc) to evaluate the model performance.  I have seen nested cross validation (whether repeated or stratified) has been used in the setting of small dataset, i.e., to generate generalized model performance while optimizing parameters selection. My question is, how can I obtain the best hyperparameters in nested cross validation (repeated/no repeated)?  I am interested in doing this in scikit-learn, if possible. I am a bit confused about how to do it.  I have read several resources but none gave me the definitive answer for this question:
19394
I am building a classification model (let's say logistic regression) using 2 different data sets. The data sets do not come from the same population. Let's say the data sets comes from different space-time coordinates (Asia in Fall vs North America in Winter). I have been told that simply mixing the data sets into one and applying a classification model is not right (closest thing I understood was concerns around bias). Are there any techniques or models that I may use to analyze these 2 data sets as one? Note that the variables/features all align (including the target variable, which is binary).
60782
I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic data. I built a Random Forest Classifier with this original dataset. I got an F1 score of 0 for the positive class. Zero precision. Zero recall. I cross-checked the predictions and test data. The model predicts some positives none of which are actually positive. Worst performance.  So, I tried to oversample the positive class. I upsampled the positives to 1000 samples. To my surprise, the F1 score for this dataset was 0.97, for the positive class. Then I tried lesser samples. I was able to achieve an F1 score of 0.83 with 200 positive samples, which is just 2.25 times of the original positive samples.  I would like to know why this occurs. For 88 samples, F1 score is 0.00 (rounded off to two digits). For 200 samples it's 0.83. There is no data leakage. All the features are engineered. I used imbalanced-learn module for oversampling. Can someone explain why is this difference in performance?
87872
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.  My question is, are there any rules of thumb that tell us when we should subsample the large category in order to force some kind of balancing in the dataset.  Examples:   If the number of positive examples is 1,000 and the number of negative examples is 10,000, should I go for training my classifier on the full dataset or I should subsample the negative examples? The same question for 1,000 positive example and 100,000 negative. The same question for 10,000 positive and 1,000 negative. etc...
104639
I searched on topics of the bias and variance trade-off and got back lots of questions with different levels of response. The information is scattering too much and unsystematic to answer my own question which stuck me. So I open a new question here and would like to hackle how I generally run a machine learning modelling process and ask for help on the issue of bias-variance assessment where I was confused or had no idea. If anyone noticed any useful posts to my questions please help point out the links and many thanks.  I execute the process basically according to the instruction below:    Question 1: When I start at step one, I try to think an issue that all my following analysis will be based on this train-test split. How about another split and is it possible to cause an obvious change in analysis results (final models, performance metrics, important features etc.)? If so, that means the analysis was split-dependent.   I have some thoughts here: assuming there exists an unknown underlying distribution of the domain set X (the whole dataset except the target class y), if the sample distribution of training set S was approximately similar to that of test set T, then the learning process should reach a good performance. Otherwise, if the two sample distributions differed in some non-trivial way, then the learner/classifier from training set S may not probably do a good job on test set T.     I made a trial to execute a 10-fold cross validation on the whole dataset and to see how variable the ROC AUC score was. In my particular dataset, I got a set of 10 scores with large variation (from maximum 0.85 to minimum 0.59). This led to my worry whether the current split I adopted for analysis was poor one.  To expand Question 1, I would like to ask further: what is the topic of this issue in machine learning theory? Is it the model stability or model variance?  I tuned the hyper-parameter/parameters in step 2 via cross validation. I try to use validation curve (refer to ) as below to choose the parameter value and avoid under/over-fitting problems.     Question 2: If this curve worked well in detecting under/over-fitting problems, what metric should be used: accuracy score/error rate, presicion, recall, f1 or roc_auc? (My current project was a classification problem with an imbalanced binary target class.)  According to the definition of bias, average error rate could be the measure of bias, so accuracy score/error rate seemed more reasonable in bias/variance issue. In fact, I concerned roc_auc score more than accuracy itself.   Question 3: how can we measure the variance? I saw a plot  and the gap was said to be able to indicate the potential variance (poorer performance on validation set than that on training set). Any other (better) way/metric can be used for this point?    In my analysis, the bias/variance check was run in the parameter tuning stage which seemed to return me a candidate model with a balanced bias/variance combination.  Question 4: does my bias/variance analysis miss some steps and how do I adjust it?  many ensemble learning methods have their own way to reduce variance or bias and  I hope to objectively evaluate the bias/variance for each candidate model.   Thanks for reading my long question but it really help for me to deliver a professional practice in my daily work. Many thanks again.
104906
I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
123279
The  for cross-validation says the following about using feature-scaling and cross-validation:      Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction   I understand the reason behind this is to prevent information leakage between the training &amp; test sets during cross-validation, which could result in an optimistic estimate of model performance.  I am wondering then, if I wish to use Principal Component Analysis to reduce the size of a feature set before training say a regression model, and PCA requires feature-scaling to be effective, how do I chain feature-scaling to PCA to cross-validated regression, without introducing data leakage between the train-test splits in the cross-validation?
48982
I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than imbalance in training data.  In other words, only changing the distribution of positive and negative classes in the test data, the AUC value may not change much. But if we change the distribution in the training data, the AUC value may largely change. The reason is that the classifier cannot be learned well. In this case, we have to use undersampling and oversampling. Am I right? I just want to make sure my understanding on AUC is correct.
69363
Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the model?  I know there is no black and white answer to this, but it would be interesting to know  some "rules of thumb" or usually used ratios. I know back at university, one of our professors used to say model on 60% and validate on 40%.