I’m dealing with a highly unbalanced dataset where 20% of data belongs to class A and 80% belongs to class B.  It’s very hard for us to produce synthetic class A data.  Just wondering if the below approach is a sensible thing to do:  Total data points:  100  Class A :  20  Class B :  80  How about splitting the dataset into 4 separate samples consisting of 20 A’s and 20 B’s. In other words, I’m mixing the 20 A’s with different samples of 20 B’s.  We’d have 4 models (say random forest or so) and finally the decision is taken from what the majority of these 4 models predict?
7818
We already had multiple questions about unbalanced data when using , , ,  and a number of other similar questions, what makes it a very popular topic! Unfortunately, each of the questions seems to be algorithm-specific and I didn't find any general guidelines for dealing with unbalanced data.  Quoting , dealing with unbalanced data     (...) heavily depends on the learning method. Most general purpose   approaches have one (or several) ways to deal with this.   But when exactly should we worry about unbalanced data? Which algorithms are mostly affected by it and which are able to deal with it? Which algorithms would need us to balance the data? I am aware that discussing each of the algorithms would be impossible on Q&amp;A site like this, I am rather looking for a general guidelines on when it could be a problem.
32233
I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max 20-30 for later use with model training (binary classification - increase / decrease).  I was thinking about using random forests for feature ranking. Is it a good idea to use them recursively? For example, let's say in the first round I drop the worst 20%, second too and so on until I get the desired number of features.  Should I use cross-validation with RF? (It's intuitive for me not to use CV because that's pretty much what RF does already.)  Also if I go with random forests should I use them as classifiers for the binary or regressor for the actual increase / decrease to get feature importances?  By the way, the models I would like to try after feature selection are: SVM, neural nets, locally weighted regressions, and random forest. I'm mainly working in Python.
92944
I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data.  I understand that this could be useful if you're working with a binary (as opposed to probabilistic or score-based) classifier and treating it as a black box, so sampling schemes are your only way to tweak its position on the "ROC curve" (in quotes because if your classifier is inherently binary I guess it doesn't have a real ROC curve, but the same concept of trading off false positives and false negatives still applies).  But it seems like the same justification doesn't hold if you actually have access to some kind of score that you're later thresholding to make a decision. In this case, isn't up-sampling just an ad-hoc way of expressing a view about your desired trade-off between false positives and false negatives when you have much better tools available, like actual ROC analysis? It seems like it would be weird in this case to expect up-sampling or down-sampling to do anything but change your classifier's "prior" on each class (i.e. unconditional probability of being that class, the baseline prediction)--I wouldn't expect it to change the classifier's "odds ratio" (how much the classifier adjusts its baseline prediction based on the covariates).  So my question is: if you have a classifier that isn't a binary black box, are there any reasons to expect up- or down-sampling to have a much better effect than adjusting the threshold to your liking? Failing that, are there any empirical studies showing reasonably large effects for up- or down-sampling on reasonable performance metrics (e.g., not accuracy)?
99109
I am a beginner in Machine Learning and I have a situation where an Image needs to be classified into first a super class, then a subclass. For Example, I have a set of images of fruits, containing images of Mangoes, Banana, Peaches, Apples etc. Then each fruit can either be fresh or rotten. I have training set containing images of rotten and fresh fruits for each Fruit type. So for a test image I need to classify which fruit it is (Mango, Banana, etc) and then whether it is fresh or rotten.  I want to know how to approach this kind of problem
82052
Lets say my multinomial logistic regression predict that a chance of a sample belonging to a each class is A=0.6, B=0.3, C=0.1 How do I threshold this values to get just binary prediction of a sample belonging to a class, taking in to an account imbalances of classes. I know what I would do if it's just a binary decision (threshold based on classes prevalence), or if the classes are balanced (classify to a class with highest probability). My end goal is to get 3x3 confusion matrix
99247
Is there an existing methodology for applying a supervised learning model to a uncertain dataset? For example, say we have a dataset with classes A and B:  +----------+----------+-------+-----------+ | FeatureA | FeatureB | Label | Certainty | +----------+----------+-------+-----------+ |        2 |        3 | A     | 50%       | |        3 |        1 | B     | 80%       | |        1 |        1 | A     | 100%      | +----------+----------+-------+-----------+   How could we train a machine learning model on this? Thanks.
53361
I've been trying to perform a binary classification using an SVM classifier (scikit-learn's SVC with RBF kernel). I have a sample size of about 100, with about 70 features each. The features are of approximately the same order of magnitude in their raw form, and the values tend to be already distributed around the 0 (not always though). The distribution of two such features is shown in the histograms below.   I performed a Z-score transformation on all features, as I know this to be considered a good practice when working with multiple features in machine learning. The problem is that when I use the raw data, I always manage to get better accuracy than with the Z-scores (about 2-3%). Bear in mind that the parameters of the SVMs in each case are optimized using a grid-search, so I'm not using exactly identical classifiers.  Does this make sense, getting worse results with Z-scores? I would expect to get the same or better results. What could be the mathematical logic behind this?  Edit  To answer two frequent questions from comments and answers:   My classes are indeed distributed equally (exactly 50%/50%) I also use measures other than accuracy (AUC, F1, etc.), but having worked on this project for some time, accuracy correlates well with what I need.
71655
My samples follow heavy tail distributions. I use a process to detect and remove "extreme" samples that goes like this:   Measure mean and standard deviation of samples.  Remove samples higher than mean plus 4 standard deviations. Repeat from Step 1 - for a total of 3 times.   If there are no extreme samples then the above process removes none; if there are many then they skew mean/std.dev, however, the multiple iterations take care of that.   For my problem, the above algorithm removes 0-5% of samples and empirical tests show that it works quite well and seems quite robust.   However, is this process sound? Does it have a formal name to look it up?  Notice that, I can not just find the "optimal" number of standard deviations above which we reject samples, since this whole process needs to be automated for multiple datasets and run as part of a live system. The datasets are similar but not identical: sometimes they are small and there are no outliers (no luxury to lose samples); sometimes they are large and the number of extreme examples that need to be removed varies and can be quite high.   Data represent user actions (one variable per action type) and we have multiple sets of users we need to compare (different segmentations). However, many "users" are actually bots that typically (but not always) do the same actions multiple times (way more times than humans would do). It is fair to say that we have a mixture of two distributions that we can't really distinguish, human-users and bots. We care about differences of behaviours of human-users after we remove as many bots as possible. In the long term we will build a classifier for bots but for now we are after some quick solution.
104639
I searched on topics of the bias and variance trade-off and got back lots of questions with different levels of response. The information is scattering too much and unsystematic to answer my own question which stuck me. So I open a new question here and would like to hackle how I generally run a machine learning modelling process and ask for help on the issue of bias-variance assessment where I was confused or had no idea. If anyone noticed any useful posts to my questions please help point out the links and many thanks.  I execute the process basically according to the instruction below:    Question 1: When I start at step one, I try to think an issue that all my following analysis will be based on this train-test split. How about another split and is it possible to cause an obvious change in analysis results (final models, performance metrics, important features etc.)? If so, that means the analysis was split-dependent.   I have some thoughts here: assuming there exists an unknown underlying distribution of the domain set X (the whole dataset except the target class y), if the sample distribution of training set S was approximately similar to that of test set T, then the learning process should reach a good performance. Otherwise, if the two sample distributions differed in some non-trivial way, then the learner/classifier from training set S may not probably do a good job on test set T.     I made a trial to execute a 10-fold cross validation on the whole dataset and to see how variable the ROC AUC score was. In my particular dataset, I got a set of 10 scores with large variation (from maximum 0.85 to minimum 0.59). This led to my worry whether the current split I adopted for analysis was poor one.  To expand Question 1, I would like to ask further: what is the topic of this issue in machine learning theory? Is it the model stability or model variance?  I tuned the hyper-parameter/parameters in step 2 via cross validation. I try to use validation curve (refer to ) as below to choose the parameter value and avoid under/over-fitting problems.     Question 2: If this curve worked well in detecting under/over-fitting problems, what metric should be used: accuracy score/error rate, presicion, recall, f1 or roc_auc? (My current project was a classification problem with an imbalanced binary target class.)  According to the definition of bias, average error rate could be the measure of bias, so accuracy score/error rate seemed more reasonable in bias/variance issue. In fact, I concerned roc_auc score more than accuracy itself.   Question 3: how can we measure the variance? I saw a plot  and the gap was said to be able to indicate the potential variance (poorer performance on validation set than that on training set). Any other (better) way/metric can be used for this point?    In my analysis, the bias/variance check was run in the parameter tuning stage which seemed to return me a candidate model with a balanced bias/variance combination.  Question 4: does my bias/variance analysis miss some steps and how do I adjust it?  many ensemble learning methods have their own way to reduce variance or bias and  I hope to objectively evaluate the bias/variance for each candidate model.   Thanks for reading my long question but it really help for me to deliver a professional practice in my daily work. Many thanks again.
19398
I have two groups of independent samples whose means I want to compare. The data comes from a distinctly non-normal distribution. (Unimodal, but heavily right skewed).    Let's say that one group (Group A) has 14,000 observations and that another group (Group B) has 700 observations. Thus each group has a decent amount of observations (we are not talking about 2000 versus 5 here).   I am of the opinion that this strong unbalance in the group sizes will not make the Mann-Whitney-Wilcoxon Test any less applicable, however a colleague is suggesting that we randomly sample 700 observations from Group A so that our group sizes are perfectly balanced. He feels that the unbalance of the groups is making the test fail to find the "correct" result. (Where correct result means "result he wants," i.e. that the two groups are statistically significantly different).   As I said, I think that having highly unequal groups does not cause the Mann-Whitney-Wilcoxon Test to somehow stop working.   As far as I understand it, I do believe that the size of the smaller group will have implications for the power of our test, but even there, I am thinking that we probably have at least a decently sufficient sample size for Group B.   Will this highly unbalanced data somehow invalidate my analysis?
108042
I got a dataset with about 5,000 columns and about 135,000 rows - all fields contain boolean (binary) data. I am looking to classify each of these columns into one of 50 groups, based on similarity.  Can anyone point me in the right direction where to start? Will a k-mean style algorithm suit me? Any library in R that I can easily access?  Note:  data is quite asymmetric in that about 95% of values are 0, 5% are 1.
107838
In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am have 5 rows (observations), and the values for Wind are [high,low,high,medium,medium].  Can I feed this categorical data into a decision tree classifier (like scikit-learn), will it take the data? That is, will the splitting of node based on wind happen on categorical values high with count 2, medium with count 2 and low with count 1. Or, will the system converts it into dummy variable?
47046
I'm working on movie recommendation algorithm. The data set consists of about 40 million ratings (user, film, rating). I want to separate the ratings into two groups - training set and probe set. The training set will be used for training, probe set for checking that the algorithm doesn't overfit the data.  My first plan was to try to follow this approach: . They create the probe set by taking the last 3 ratings from every user (and every user is guaranteed to have 20+ ratings). But the problem is, what if there is some really new film whose all ratings are in the probe set? The training algorithm wouldn't have any info about this film.
90412
I need to analyse data and am unsure whether or not I should use a multinomial logistic regression analysis or a binary logistic regression analysis. I have just conducted a latent class analysis and am interested in seeing if my variables can predict class membership. There are only two classes and so my response variable only has two categories; however the variables range from having 2 to 3 to 4 and are mostly nominal categorical data.
25641
I know that a neural-network architecture is mostly based on the problem itself and the types of input/output, but still - there's always a "square one" when starting to build one. So my question is - given a input dataset of MxN (M  is the number of records, N is the number of features) and a C possible output classes - is there a thumb-rule to how many layers/units should we start with?
59110
Let's say you have a large random sample of soccer players in Europe but you are only interested in what happens in Spain. Could you reduce your sample to players in Spain and still call it a random sample (but of a different population)? If not, how would you call that subsample and which specific precautions should you take to be able to make inferences on the population of spanish soccer players?  My feeling is that using that subsample would be fine as long as it is large enough, but maybe I am missing something.
64236
I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are:   SMOTE: Synthetic Minority Over-sampling TEchnique () ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning ()   I have implemented ADASYN because its adaptive nature and ease to extension to multi-class problems.  My question is how to test the oversampled data produced by ADASYN (or any other oversampling methods). It is not clear in the mentioned two paper how they performed their experiments. There are two scenarios:  1- Oversample the whole dataset, then split it to training and testing sets (or cross validation).  2- After splitting the original dataset, perform oversampling on the training set only and test on the original data test set (could be performed with cross validation).  In the first case the results are much better than without oversampling, but I am concerned if there is overfitting. While in the second case the results are slightly better than without oversampling and much worse than the first case. But the concern with the second case is if all minority class samples goes to the testing set, then no benefit will be achieved with oversampling.  I am not sure if there are any other settings to test such data.