I’m currently encountering some problems analyzing a dataset with neural network. The problem is that I have an unbalanced binary class training set (10:1). Training accuracy for both classes are 100%. When predicting I get accuracy of 99% for the majority class in the validation set, and accuracy of 0.2% for the minority class. So I was wondering if over-sampling would help to improve the accuracy for the minority class.
46805
This is a question in general, not specific to any method or data set. How do we deal with a class imbalance problem in Supervised Machine learning where the number of 0 is around 90% and number of 1 is around 10% in your dataset.How do we optimally train the classifier.  One of the ways which I follow is sampling to make the dataset balanced and then train the classifier and repeat this for multiple samples.   I feel this is random, Is there any framework to approach these kind of problems.
15961
I want to know what it is, and how it is any different from ensembling?  Suppose, I want to achieve high accuracy in classification and segmentation, for a specific task, if I use different networks, such as CNN, RNN, etc to achieve this, is this called an end to end model? (architecture?) or not?
30145
See also .   In  algorithms such as  and  it is known that the "weak" learners to be combined only have to perform better than chance to be useful, from Wikipedia:     The classifiers it uses can be weak (i.e., display a substantial error rate), but as long as their performance is not random (resulting in an error rate of 0.5 for binary classification), they will improve the final model. Even classifiers with an error rate higher than would be expected from a random classifier will be useful, since they will have negative coefficients in the final linear combination of classifiers and hence behave like their inverses.    What are the benefits of using weak as opposed to strong learners? (e.g. why not boost with "strong" learning methods - are we more prone to overfitting?) Is there some sort of "optimal" strength for the weak learners? And is this related to the number of learners in the ensemble?   Is there any theory to back up the answers to these questions?
6455
I am trying to classify images to more then a 100 classes, of different sizes ranged from 300 to 4000 (mean size 1500 with std 600). I am using a pretty standard CNN where the last layer outputs a vector of length number of classes, and using pytorch's loss function CrossEntropyLoss.  I tried to use $weights = \frac{max(sizes)}{sizes}$ for the cross entropy loss which improved the unweighted version, not by much.  I also thought about duplicating images such that all classes ends up to be of the same size as the larges one.  Is there any standard way of handling this sort of imbalance?
45382
I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).  Why is this a good idea?
121830
I want to compare the classification performance of two heuristics: $h_0$ and $h_1$. I have around 700 samples for which I know the output class $c$ and the results of the two heuristics $c_{\textit{h_0}}, c_{\textit{h_1}}$.  That is, for each sample I have $\text{sample}_{\textit{id}}, c, c_{\textit{h_0}}, c_{\textit{h_1}}$.   $h_0$ has around ~71% correct predictions of the output class and $h_1$ has around ~77% correct. What I want to calculate now is whether this performance difference is just by chance or if the prediction performance of $h_1$ is statistical significant better than that of $h_0$.  I have found , but I don't know whether I can just put my data in there like the following matrix to calculate a p-value.  $$   \begin{matrix}                                &amp; h_0 &amp; h_1 \\     \text{correct prediction} &amp; 497 &amp; 539 \\     \text{wrong prediction}   &amp; 203 &amp; 161   \end{matrix} $$  I don't know if I can calculate the p-value like that or if I'm completely on the wrong track. I would really appreciate it if someone could point me in the right direction.
49420
I hope the following excerpts will provide an insight into what my question is going to be. These are from .  The learning then gradually slows down. Finally, at around epoch 280 the classification accuracy pretty much stops improving. Later epochs merely see small stochastic fluctuations near the value of the accuracy at epoch 280. Contrast this with the earlier graph, where the cost associated to the training data continues to smoothly drop. If we just look at that cost, it appears that our model is still getting &quot;better&quot;. But the test accuracy results show the improvement is an illusion. Just like the model that Fermi disliked, what our network learns after epoch 280 no longer generalizes to the test data. And so it's not useful learning. We say the network is overfitting or overtraining beyond epoch 280.  We are training a neural network and the cost (on training data) is dropping till epoch 400 but the classification accuracy is becoming static (barring a few stochastic fluctuations) after epoch 280 so we conclude that model is overfitting on training data post epoch 280.  We can see that the cost on the test data improves until around epoch 15, but after that it actually starts to get worse, even though the cost on the training data is continuing to get better. This is another sign that our model is overfitting. It poses a puzzle, though, which is whether we should regard epoch 15 or epoch 280 as the point at which overfitting is coming to dominate learning? From a practical point of view, what we really care about is improving classification accuracy on the test data, while the cost on the test data is no more than a proxy for classification accuracy. And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.  As opposed to classification accuracy on test data compared with training cost previously we are now placing cost on test data against training cost. Then the book goes on to explain why 280 is the right epoch where the overfitting has started. That is what I have an issue with. I can't wrap my head around this. We are asking model to minimize the cost and thus cost is the metric it uses as a measure of its own strength to classify correctly. If we consider 280 as the right epoch where the overfitting has started, have we not in a way created a biased model that though is a  better classifier on the particular test data but nonetheless is making decisions with low confidence and hence is more prone to deviate from the results shown on the test data?
117577
I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc)  In many cases hyperparameter tuning on the whole dataset using either Grid or Random search is too costly time-wise.  I started using the following technique   Sub sample my dataset Use the obtained fraction to tune the hyperparameters on Use the obtained parameters to train a model using the whole dataset   To evaluate each set of parameters on the second step I use sklearn's GridSearchCV with cv=10. To evaluate the final model that I create in the third step I use sklearn's cross_val_predict. In that sense I evaluate my models leaving a 10% percent of data out, I train on the rest and measure the predictive accuracy on the 10%, iteratively 10 times, then taking the average of the scores.  What made me worry is that the prediction accuracy I get from training on my whole dataset, is really close to the evaluation I get when tuning the parameters for the best set of parameters (each tested set of parameters outputs a score obtained from averaging 10-fold-cross validation results).  Most of the times the accuracy that cross_val_predict measured using all the training examples (whole dataset) is a little bit above what the evaluation of the best parameters returned.  To illustrate this here is the evaluation of a set of parameters (on a smaller dataset than what I described above but the effect is the same)  Best parameters set found on development set: {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} Scores for all sets of parameters 0.851 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.5} 0.852 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.1} 0.829 (+/-0.006) for {'kernel': 'rbf', 'C': 3, 'gamma': 0.001} 0.853 (+/-0.006) for {'kernel': 'rbf', 'C': 9, 'gamma': 0.1} ...   And here are the averaged scores (from cross_val_predict) I got from training on my whole dataset using the best parameters  precision    recall  f1-score   support        0       0.86      0.85      0.86     15417       1       0.86      0.87      0.87     16561  avg / total       0.86      0.86      0.86     31978  acc score: 0.863750078179 roc au score: 0.863370490059 [[13147  2270]  [ 2087 14474]]   As you can see training on the whole dataset improves the results. I have also validated that badly tuned model (e.g. using the default values or random values for C and gamma) leads to much worse prediction accuracy.   Overall I think that tuning the hyperparameters on a subset is not ideal but can potentially lead to relatively good results without having to wait too long. I for example before using that approach used optunity package for tuning the hyperparameter on the whole dataset. This procedure would take 3-5 days to complete and would produce results that either had really good precision or really good recall but not both, so although for each class either the precision or the recall was really high (higher than what any of my other classifiers had achieved) the f1 meassure was really low. In the contrary using the later approach leads to some hours of training and a better f1 meassure.   My concerns are:  Do I limit my classification accuracy? Do I avoid using all the prediction power that my dataset can offer by tuning only on a subset? If such a harm of performance is happening is it somehow limited by some factor?
5573
First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only for predictions with only one class? If I were to use a categorical cross-entropy loss, which is typically found in most libraries (like TensorFlow), would there be a significant difference?  In fact, what are the exact differences between a categorical and binary cross-entropy? I have never seen an implementation of binary cross-entropy in TensorFlow, so I thought perhaps the categorical one works just as fine.
77787
When should we discretize/bin independent variables/features and when should not?  My attempts to answer the question:   In general, we should not bin, because binning will lose information. Binning is actually increasing the degree of freedom of the model, so, it is possible to cause over-fitting after binning. If we have a "high bias" model, binning may not be bad, but if we have a "high variance" model, we should avoid binning. It depends on what model we are using. If it is a linear mode, and data has a lot of "outliers" binning probability is better. If we have a tree model, then, outlier and binning will make too much difference.   Am I right? and what else?    I thought this question should be asked many times but I cannot find it in CV only these posts
19398
I have two groups of independent samples whose means I want to compare. The data comes from a distinctly non-normal distribution. (Unimodal, but heavily right skewed).    Let's say that one group (Group A) has 14,000 observations and that another group (Group B) has 700 observations. Thus each group has a decent amount of observations (we are not talking about 2000 versus 5 here).   I am of the opinion that this strong unbalance in the group sizes will not make the Mann-Whitney-Wilcoxon Test any less applicable, however a colleague is suggesting that we randomly sample 700 observations from Group A so that our group sizes are perfectly balanced. He feels that the unbalance of the groups is making the test fail to find the "correct" result. (Where correct result means "result he wants," i.e. that the two groups are statistically significantly different).   As I said, I think that having highly unequal groups does not cause the Mann-Whitney-Wilcoxon Test to somehow stop working.   As far as I understand it, I do believe that the size of the smaller group will have implications for the power of our test, but even there, I am thinking that we probably have at least a decently sufficient sample size for Group B.   Will this highly unbalanced data somehow invalidate my analysis?
47279
When dealing with the class imbalance problem in a binary classifier, there are three ways I know of to address it: over-sampling, under-sampling and using cost-sensitive methods.   Are there any guidelines, rules of thumb or general strategies to choose among these methods? A possible answer would be: over-sample when the positive class has more than 100 instances (I just made it up).
47125
Recently, I built a classification model based on the imbalanced data set(positive sample is minority and negative sample is majority), and the model gave the following result for the test set:  True Positives = 0 True Negatives = 139 False Positives = 0 False Negatives = 10.  My question is: for the result, can Matthews correlation coefficient ( ) and  be used for estimating the classifier? Since the denominators for MCC and F-measure are zero, it seems meaningless. If so, MCC and F-measure is not always works for estimating the classifier, and sensitivity and specificity as well as g-mean should be better. Is that right? Any help is appreciated.
85510
So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would compare the usefulness of each combination.  For example I can both train an SVM using Features 1, 10, 15 to get 100% accuracy. But at the same time I can train a logistic regression classifier only using Feature 7 to get 100% accuracy. Also this is a binary classification problem.
38598
I'm working on a project using logistic regression to predict student retention. The data were collected through three self-report instruments. We are trying to find out which predictors are powerful enough to predict the at-risk students. I came across some articles saying a balanced sample (50% stay, 50% dropouts) is desirable for such study, e.g. Glynn, J.G., Sauer, P.L., &amp; Miller, T.E. (2003). , NASPA Journal, 41 (1), 41-67:     A problem however, is that the distribution of the dependent variable is likely to be highly skewed toward persistence. For example, if 85% of the analysis sample were persistors, a classification model that classified every student as a persistor would have a success rate of 85%, or would classify 85% of student correctly. To resolve this issue, the maintenance of relative balance between the number of dropouts and the number of persistors (about 50% each) in the analysis sample was desirable.   Is this true? Our sample only has about 25%-30% dropout students. Will this affect the results?
93962
Is it possible to have test error lower than training error?  I have a classification problem with 2000 samples, 500 of which are positives, 1500 are negatives. I split my data into 70% training data, 30% test data.  Run random forest with 200 estimators and cv=10. I did this several times and compared the recall and precision score and notice the scores for my test set is significantly better. Is this possible?
16634
I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar architectures and data. How can I fix this?    As for question        to which this question is inspired, the question is intentionally left general so that other questions about how to reduce the generalization error of a neural network down to a level which has been proved to be attainable, can be closed as duplicates of this one.  See also dedicated thread on Meta: