I'm working with unbalanced class databases and I'm wondering what's the best practice to retrieve the best threshold cutoff value using roc curve ?  I want the best cutoff to maximize my F1 score.      I already have a roc curve I'm wondering what the best practice to   generate the best threshold cutoff.
56168
I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9):    My questions are:    How to determine cutoff point for this test and its confidence interval where readings should be judged as ambiguous? What is the best way to visualize this (using ggplot2)?   Graph is rendered using ROCR and ggplot2 packages:  #install.packages("ggplot2","ROCR","verification") #if not installed yet library("ggplot2") library("ROCR") library("verification") d &lt;-read.csv2("data.csv", sep=";") pred &lt;- with(d,prediction(x,test)) perf &lt;- performance(pred,"tpr", "fpr") auc &lt;-performance(pred, measure = "auc")@y.values[[1]] rd &lt;- data.frame(x=perf@x.values[[1]],y=perf@y.values[[1]]) p &lt;- ggplot(rd,aes(x=x,y=y)) + geom_path(size=1) p &lt;- p + geom_segment(aes(x=0,y=0,xend=1,yend=1),colour="black",linetype= 2) p &lt;- p + geom_text(aes(x=1, y= 0, hjust=1, vjust=0, label=paste(sep = "", "AUC = ",round(auc,3) )),colour="black",size=4) p &lt;- p + scale_x_continuous(name= "False positive rate") p &lt;- p + scale_y_continuous(name= "True positive rate") p &lt;- p + opts(             axis.text.x = theme_text(size = 10),             axis.text.y = theme_text(size = 10),             axis.title.x = theme_text(size = 12,face = "italic"),             axis.title.y = theme_text(size = 12,face = "italic",angle=90),             legend.position = "none",             legend.title = theme_blank(),             panel.background = theme_blank(),             panel.grid.minor = theme_blank(),              panel.grid.major = theme_line(colour='grey'),             plot.background = theme_blank()             ) p   data.csv contains the following data:  x;group;order;test 56;Tumor;1;1 55;Tumor;1;1 52;Tumor;1;1 60;Tumor;1;1 54;Tumor;1;1 43;Tumor;1;1 52;Tumor;1;1 57;Tumor;1;1 50;Tumor;1;1 34;Tumor;1;1 24;Normal;2;0 34;Normal;2;0 22;Normal;2;0 32;Normal;2;0 25;Normal;2;0 23;Normal;2;0 23;Normal;2;0 19;Normal;2;0 56;Normal;2;0 44;Normal;2;0
60782
I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic data. I built a Random Forest Classifier with this original dataset. I got an F1 score of 0 for the positive class. Zero precision. Zero recall. I cross-checked the predictions and test data. The model predicts some positives none of which are actually positive. Worst performance.  So, I tried to oversample the positive class. I upsampled the positives to 1000 samples. To my surprise, the F1 score for this dataset was 0.97, for the positive class. Then I tried lesser samples. I was able to achieve an F1 score of 0.83 with 200 positive samples, which is just 2.25 times of the original positive samples.  I would like to know why this occurs. For 88 samples, F1 score is 0.00 (rounded off to two digits). For 200 samples it's 0.83. There is no data leakage. All the features are engineered. I used imbalanced-learn module for oversampling. Can someone explain why is this difference in performance?
20081
I have a test set of 100 cases and two classifiers.  I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers.  Question 1: How can I compute p-value to check if one is significantly better than the other with respect to all scores (ROC AUC, sensitivity, specificity)?    Now, for the same test set of 100 cases, I have different and independent feature assignments for each case. This is because my features are fixed but subjective and provided by multiple (5) subjects.  So, I evaluated my two classifiers again for 5 "versions" of my test set and obtained 5 ROC AUCs, 5 sensitivities and 5 specificities for both classifiers. Then, I computed the mean of each performance measure for 5 subjects (mean ROC AUC, mean sensitivity and mean specificity) for both classifiers.  Question 2: How can I compute p-value to check if one is significantly better than the other with respect to mean scores (mean ROC AUC, mean sensitivity, mean specificity)?    Answers with some example python (preferably) or MatLab code are more than welcome.
113895
What decides the choice of function ( Softmax vs Sigmoid ) in a Logistic classifier ?  Suppose there are 4 output classes . Each of the above function gives the probabilities of each class being the correct output . So which one to take for a classifier ?
100975
In a recent, well recieved, question, Tim asks ?  The premise of the question is that there is a lot of machine learning literature discussing class balance and the problem of imbalanced classes.  The idea is that datasets with an imbalance between the positive and negative class cause problems for some machine learning classification (I'm including probabilistic models here) algorithms, and methods should be sought to "balance" the dataset, restoring the perfect 50/50 split between positive and negative classes.  The general sense of the upvoted answers is that "it's not, at least if you are thoughtful in your modeling".  M. Henry L., in an up-voted comment to an accepted answer, states     [...] there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.   AdamO argues that the "problem" with class balance is really one of class rarity     Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have small sample size. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.   If this is the true issue at hand, it leaves an open question: what is the purpose of all the resampling methods intended to balance the dataset: oversampling, undersampling, SMOTE, etc?  Clearly they don't address the problem of implicitly having a small sample size, you can't create information out of nothing!
48982
I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than imbalance in training data.  In other words, only changing the distribution of positive and negative classes in the test data, the AUC value may not change much. But if we change the distribution in the training data, the AUC value may largely change. The reason is that the classifier cannot be learned well. In this case, we have to use undersampling and oversampling. Am I right? I just want to make sure my understanding on AUC is correct.
93095
I've been thinking a lot about the &quot;class imbalance problem&quot; in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on. First let me define (or attempt to) define my terms:  The class imbalance problem in machine/statistical learning is the observation that some binary classification(*) algorithms do not perform well when the proportion of 0 classes to 1 classes is very skewed.  So, in the above, for example, if there were one-hundred $0$ classes for every single $1$ class, I would say the class imbalance is $1$ to $100$, or $1\%$. Most statements of the problem I have seen lack what I would think of as sufficient qualification (what models struggle, how imbalanced is a problem), and this is one source of my confusion. A survey of the standard texts in machine/statistical learning turns up little:  Elements of Statistical Leaning and Introduction to Statistical Learning do not contain &quot;class imbalance&quot; in the index.  Machine Learning for Predictive Data Analytics also does not contain&quot;class imbalance&quot; in the index.  Murphy's Machine Learning: A Probabilistic Perspective does contain &quot;class imbalance* in the index.  The reference is to a section on SVM's, where I found the following tantalizing comment:  It is worth remembering that all these difficulties, and the plethora of heuristics that have been proposed to fix them, fundamentally arise because SVM's do not model uncertainty using probabilities, so their output scores are not comparable across classes.    This comment does jive with my intuition and experience: at my previous job we would routinely fit logistic regressions and gradient boosted tree models (to minimize binomial log-likelihood) to unbalanced data (on the order of a $1\%$ class imbalance), with no obvious issues in performance. I have read (somewhere) that classification tree based models (trees themselves and random forest) do also suffer from the class imbalance problem.  This muddies the waters a little bit, trees do, in some sense, return probabilities: the voting record for the target class in each terminal node of the tree. So, to wrap up, what I'm really after is a conceptual understanding of the forces that lead to the class imbalance problem (if it exists).  Is it something we do to ourselves with badly chosen algorithms and lazy default classification thresholds? Does it vanish if we always fit probability models that optimize proper scoring criteria?  Said differently, is the cause simply a poor choice of loss function, i.e. evaluating the predictive power of a model based on hard classification rules and overall accuracy? If so, are models that do not optimize proper scoring rules then useless (or at least less useful)?  (*) By classification I mean any statistical model fit to binary response data.  I am not assuming that my goal is a hard assignment to one class or the other, though it may be.
53361
I've been trying to perform a binary classification using an SVM classifier (scikit-learn's SVC with RBF kernel). I have a sample size of about 100, with about 70 features each. The features are of approximately the same order of magnitude in their raw form, and the values tend to be already distributed around the 0 (not always though). The distribution of two such features is shown in the histograms below.   I performed a Z-score transformation on all features, as I know this to be considered a good practice when working with multiple features in machine learning. The problem is that when I use the raw data, I always manage to get better accuracy than with the Z-scores (about 2-3%). Bear in mind that the parameters of the SVMs in each case are optimized using a grid-search, so I'm not using exactly identical classifiers.  Does this make sense, getting worse results with Z-scores? I would expect to get the same or better results. What could be the mathematical logic behind this?  Edit  To answer two frequent questions from comments and answers:   My classes are indeed distributed equally (exactly 50%/50%) I also use measures other than accuracy (AUC, F1, etc.), but having worked on this project for some time, accuracy correlates well with what I need.
117453
I've applied 6 classifiers on my data set using 5-fold CV to calculate average AUC for each model. What method do you suggest me to determine which model has the highest AUC? I've heard of t-test and Mann Whitney test. But I don't know if using these tests are useful or should I select the best model simply by considering the model with the highest AUC by looking at them?
103534
I'm trying to solve  from the UCI database repository. Unfortunately (or fortunately), I've noticed that my dataset is imbalanced. I've structured the data as 5 classes, according to the final mark reached by the pupil, like so:   If student gets a mark from 0 to 7 => class 1 [FAIL(E)] If student gets a mark from 8 to 9 => class 2 [SUFFICIENT(D)] If student gets a mark from 10 to 11 => class 3 [GOOD(C)] If student gets a mark from 12 to 15 => class 4 [NOTABLE(B)] If student gets a mark from 16 to 19 => class 5 [OUTSTANDING(A)]   My problem is that, as I said, data are imbalanced, so I want to balance it.  I've thought about applying some kind of undersampling method, but my dataset has only 649 instances so I think removing some of them is not the best idea. Then I thought about doing some oversampling, in order to replicate some of the minority class examples and then get classes balanced, but I'm still unsure if that could work.   I would be very grateful if you could give me a hand with this. Its the first time I face a real problem with imbalanced data.
74770
I applied  two different classifiers against the same validation set. It turns out that classifier A is better than classifier B in terms of ROC curve. However, classifier B is better than classifier A in terms of confusion matrix. How to explain this kind of contradiction?
17518
Scikit Learn's  mentions the use of nested cross-validation:     &gt;&gt;&gt; clf = GridSearchCV(estimator=svc, param_grid=dict(gamma=gammas),   ...                    n_jobs=-1)   &gt;&gt;&gt; cross_validation.cross_val_score(clf, X_digits, y_digits)       Two cross-validation loops are performed in parallel: one by the GridSearchCV estimator to set gamma and the other one by cross_val_score to measure the prediction performance of the estimator. The resulting scores are unbiased estimates of the prediction score on new data.   From what I understand, clf.fit will use cross-validation natively to determine the best gamma. In that case, why would we need to use nested cv as given above? The note mentions that nested cv produces "unbiased estimates" of the prediction score. Isn't that also the case with clf.fit?  Also, I was unable to get the clf best estimates from the cross_validation.cross_val_score(clf, X_digits, y_digits) procedure. Could you please advise how that can be done?
100158
I'm comparing different pokemon of the same type (e.g. comparing pidgeys with each other), to see which ones I want to keep and which to transfer. I want to select based on Max CP -which differs per each pokemon-, but it's a bitch to find what a pokémon's max CP would be.  Even though the bar fill isn't very accurate (not determined per every 1%), if a CP bar is filled to 50%, it's not hard to find out what the max CP of that pokemon is. But otherwise, since there are no round measurement tools in any software I know of, it stays at mere guessing.  Does anyone know of a way to do this effectively?
30589
I used sklearn to compute roc_auc_score for a dataset of 72 instances. The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this possible? I would think that even one misclassification should have dropped the score to slightly below 1.0.  # Python 3.6.4 # numpy==1.14.3 # scikit-learn==0.19.1 # scipy==1.1.0  from sklearn import metrics import numpy as np  y_true = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]) y_prob = np.array([0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.7, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.9, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.9, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0])  # Show which actuals do not match their expected probabilities for index, (actual, predicted_prob) in enumerate(zip(y_true, y_prob)):   if (actual == 1 and predicted_prob &lt;= 0.5) or (actual == 0 and predicted_prob &gt; 0.5):     print (f'Mismatch at index {index}. Actual={actual}, predicted_prob={predicted_prob}')   rocauc = metrics.roc_auc_score(y_true, y_prob)     print (f'ROCAUC: {rocauc}')  # Outputs: # Mismatch at index 14. Actual=1.0, predicted_prob=0.5 # Mismatch at index 68. Actual=1.0, predicted_prob=0.4 # ROCAUC: 1.0   Then I debugged the score computation itself and looked at the coordinate ROC output.  # In sklearn/metrics/ranking.py, line 271: --&gt; 271         fpr, tpr, tresholds = roc_curve(y_true, y_score,     272                                         sample_weight=sample_weight)  ipdb&gt; fpr array([0.        , 0.        , 0.        , 0.18181818, 1.        ]) ipdb&gt; tpr array([0.33333333, 0.66666667, 1.        , 1.        , 1.        ]) ipdb&gt; tresholds array([0.9, 0.7, 0.4, 0.1, 0. ])  # coords = [] # for x, y in zip(fpr, tpr): #   coords.append((x, y))  ipdb&gt; pp coords [(0.0, 0.3333333333333333),  (0.0, 0.6666666666666666),  (0.0, 1.0),  (0.18181818181818182, 1.0),  (1.0, 1.0)]   All of those coordinates are on x=0 or y=1, meaning ROCAUC is showing 1.0. The only plausible explanation I can think of is that if the alg had more fpr/tpr points, it would show a very tight curve that never reaches x,y = (0, 1), and the ROCAUC would be close to 1, but not exactly 1. Is that a reasonable interpretation or am I missing something?
121542
I trained a model with some algorithms like random forest, logistic regression and so on.   My dataset was split into 80% CV train data (so actually 60% of data to train the model and 20 % for testing with cv). I built my model and now I'm using the last 20% (which I didn't used the whole time) but I'm wondering why is the predictive accuracy better with the validation set than with the training set?    specificity validation: 0.62962963 sensitivity validation: 0.94761905   on the left is the mean, on the right is the standard deviation (+/-) with a 10 times cross validation    specificity_train: 0.55  (+/- 0.10)  recall_train: 0.94   (+/- 0.02)   Did I something wrong? Thought this is the normal / best way to test, if the model is overfitted or not.  Or is it OK due to the standard deviation?
86140
I have a very unbalanced sample set, e.g. 99% true and 1% false.   Is it reasonable to select a balanced subset with a 50/50-distribution for neural network training? The reason for this is, that I guess training on the original data set may induce a bias on the true-samples.  Can you suggest me some literature that covers this topic especially for neural networks?
3193
I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?