I have come across two different terms regarding Area Under Curve (AUC):   ROC AUC: The Area Under an ROC(Receiver operating characteristic) Curve AUPRC: The Area Under Precision-Recall Curve   Are they talking about the same things?   If not, do they share similar values for all possible datasets?  If still not, an example of dataset where ROC AUC and AUPRC strongly disagrees would be great.
27212
I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed the ROC-AUC and the Precision-Recall-AUC (PR-AUC). Then the surprise came!   Method 1 is better than method 2 when I compare them using ROC-AUC. Method 2 is better than method 1 when I compare them using PR-AUC.   Now I'm so confused! How to say which method is better? As far as I know from  that if the ROC-AUC is high, then PR-AUC is also high. So if the ROC curve of method-1 dominates, so should method-1's PR curve. Is my understanding incorrect? Or am I missing something? Because I'm really going crazy!
18830
\draw (0,0) -- (60:.75cm) arc (60:180:.75cm);   What does the colon in (60:.75cm) mean? Normally, I'd define a normal point there.
35780
I have calculated AIC and AICc to compare two general linear mixed models; The AICs are positive with model 1 having a lower AIC than model 2.  However, the values for AICc are both negative (model 1 is still &lt; model 2).  Is it valid to use and compare negative AICc values?
90700
Is it possible to overfit a logistic regression model? I saw a video saying  that if my area  under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to overfit a logistic regression model?
40623
Akaike Information Criterion (AIC) and the c-statistic (area under ROC curve) are two measures of model fit for logistic regression. I am having trouble explaining what is going on when the results of the two measures are not consistent. I guess they are measuring slightly different aspects of model fit, but what are those specific aspects?  I have 3 logistic regressions models. Model M0 has some standard covariates. Model M1 adds X1 to M0; model M2 adds X2 to M0 (so M1 and M2 are not nested).  The difference in AIC from M0 to both M1 and M2 is about 15, indicating X1 and X2 both improve model fit, and by about the same amount.  c-statistics are: M0, 0.70; M1, 0.73; M2 0.72. The difference in c-statistic from M0 to M1 is significant (method of DeLong et al 1988), but the difference from M0 to M2 is not significant, indicating that X1 improves model fit, but X2 does not.  X1 is not routinely collected. X2 is supposed to be routinely collected but is missing in about 40% of cases. We want to decide whether to start collecting X1, or improve collection of X2, or drop both variables.  From AIC we conclude that the variables make similar improvement to the model. It's probably easier to improve collection of X2 than start collecting a completely new variable (X1), so we would aim to improve X2 collection. But from c-statistic, X1 improves the model and X2 does not, so we should forget about X2 and start collecting X1.  As our recommendation depends on which statistic we focus on, we need to clearly understand the difference in what they are measuring.  Any advice welcome.
122242
What is the exact difference between an autoregressive (AR) and vector autoregressive model (VAR)?  I always thought that VAR would just be for more than two variables, until I learned that AR can also have more than two variables. I.e. I had a misunderstanding(?) that AR models are not always bivariate.  Before I write too much potentially false assumptions about the difference between AR and VAR, it would rather be interested to see if anyone can make provide a quick and good summary.
11763
I am using a classifier which returns probabilities. To calculate AUC, I am using pROC R-package. The output probabilities from classifier are:  probs=c(0.9865780, 0.9996340, 0.9516880, 0.9337157, 0.9778576, 0.8140116, 0.8971550, 0.8967585, 0.6322902, 0.7497237)   probs shows probability of being in class '1'. As shown, the classifier has classified all of samples in class '1'.  True label vector is:  truel=c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0)   As shown, classifier has misclassified 5 samples. But, AUC is:  pROC::auc(truel, probs) Area under the curve: 1   Could you please explain to me why it happens?
91936
I wanted to ask in general what area under the graph means. Also which physical quantity is highlighted by area under distance vs time graph. I'm confused that area is a 2 dimensional concept and it may indicate distance or displacement, which are 1 dimensional quantities.
85998
Why is the area under the ROC curve the probability that a classifier will rank a randomly chosen "positive" instance (from the retrieved predictions) higher than a randomly chosen "positive" one (from the original positive class)? How does one prove this statement mathematically using integral, giving the CDFs and PDFs of the true positive and negative class distributions?
16653
Something that really confuses me with how ROC plots are generated is that, according to Wikipedia:  The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.  So the algorithm seems to be: Set a decision threshold for binary classification, e.g., a number between $[0, 1]$ that tells which data is labeled as $1$ or $0$. Run the trained classifier on your dataset. Calculate TP and FP, which is just based on the result of your classification. Then plot a point with coordinate (TP, FP) on a 2D plot. Repeat this process for....(?) many many times, until you get many points. Then linearly interpolate between these points to generate a 2D curve. Do I get this process correct?  So my question is: why isn't this naturally a 3D plot? The threshold parameter seems to be pretty important.  First of all, I cannot tell at any given plot on the ROC curve what the threshold was used. Are the points even plotted according some sorted order of these thresholds? I cannot tell.  Also, shouldn't the threshold be critical at creating your final classifier? The ROC curve should have a point that corresponds to the best threshold. But I don't know what that threshold is on this plot.  Finally, this is not related to the ROC plot per-se but just the entire concept. It seems pretty dumb to me to either use a very large or very small threshold. A decision threshold of $0$ or $1$ would not make any sense in creating a classifier, no?   It just doesn't seem to make sense to neglect this information.
58293
When talking about longitudinal data, we may refer to data collected over time from the same subject / study unit repeatedly, thus there are correlations for the observations within the same subject, i.e., within-subject similarity.  When talking about time-series data, we also refer to the data collected over a series of time and it seems very similar to the longitudinal setting mentioned above.  I'm wondering if someone can provide a clear clarification between these two terms, what is the relationship and what are the differences?
9786
Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
39340
It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use one over the other. A real example would be incredibly useful.
30589
I used sklearn to compute roc_auc_score for a dataset of 72 instances. The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this possible? I would think that even one misclassification should have dropped the score to slightly below 1.0.  # Python 3.6.4 # numpy==1.14.3 # scikit-learn==0.19.1 # scipy==1.1.0  from sklearn import metrics import numpy as np  y_true = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]) y_prob = np.array([0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.7, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.9, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.9, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0])  # Show which actuals do not match their expected probabilities for index, (actual, predicted_prob) in enumerate(zip(y_true, y_prob)):   if (actual == 1 and predicted_prob &lt;= 0.5) or (actual == 0 and predicted_prob &gt; 0.5):     print (f'Mismatch at index {index}. Actual={actual}, predicted_prob={predicted_prob}')   rocauc = metrics.roc_auc_score(y_true, y_prob)     print (f'ROCAUC: {rocauc}')  # Outputs: # Mismatch at index 14. Actual=1.0, predicted_prob=0.5 # Mismatch at index 68. Actual=1.0, predicted_prob=0.4 # ROCAUC: 1.0   Then I debugged the score computation itself and looked at the coordinate ROC output.  # In sklearn/metrics/ranking.py, line 271: --&gt; 271         fpr, tpr, tresholds = roc_curve(y_true, y_score,     272                                         sample_weight=sample_weight)  ipdb&gt; fpr array([0.        , 0.        , 0.        , 0.18181818, 1.        ]) ipdb&gt; tpr array([0.33333333, 0.66666667, 1.        , 1.        , 1.        ]) ipdb&gt; tresholds array([0.9, 0.7, 0.4, 0.1, 0. ])  # coords = [] # for x, y in zip(fpr, tpr): #   coords.append((x, y))  ipdb&gt; pp coords [(0.0, 0.3333333333333333),  (0.0, 0.6666666666666666),  (0.0, 1.0),  (0.18181818181818182, 1.0),  (1.0, 1.0)]   All of those coordinates are on x=0 or y=1, meaning ROCAUC is showing 1.0. The only plausible explanation I can think of is that if the alg had more fpr/tpr points, it would show a very tight curve that never reaches x,y = (0, 1), and the ROCAUC would be close to 1, but not exactly 1. Is that a reasonable interpretation or am I missing something?
24698
What diagnostic plots (and perhaps formal tests) do you find most informative for regressions where the outcome is a count variable?  I'm especially interested in Poisson and negative binomial models, as well as zero-inflated and hurdle counterparts of each. Most of the sources I've found simply plot the residuals vs. fitted values without discussion of what these plots "should" look like.  Wisdom and references greatly appreciated. The back story on why I'm asking this, if it's relevant, is .  Related discussions:
122094
I have 100,000 observations (9 dummy indicator variables) with 1000 positives.  Logistic Regression should work fine in this case but the cutoff probability puzzles me.    In common literature, we choose 50% cutoff to predict 1s and 0s.  I cannot do this as my model gives a maximum value of ~1%.  So a threshold can be at 0.007 or somewhere around it.    I do understand ROC curves and how the area under curve can help me choose between two LR models for the same dataset.  However, ROC doesn't help me choose an optimum cutoff probability that can be used to test the model on an out-of-sample data.   Should I simply use a cutoff value that minimizes the misclassification rate? ()  Added --> For such a low event rate, my misclassificiation rates are affected by a huge number of false positives.  While the rate over all appears good as total universe size is also big, but my model should not have so many false positives (as it is an investment return model).  5/10 coeff are significant.
111760
I have a dataset with two classes of elements. I also have two methods which assign (complementary) probabilities to each element in the dataset of belonging to either class.  Given that I work with probabilities (instead of hard 0,1 classification values), I was pointed to  as a way to asses which method performs better. The two most used rules appear to be:    (Log loss, logistic loss, cross-entropy loss)  (Brier score)   with Log loss apparently being the standard approach (is it?). I also found scikit-learn's  , an implementation of the:    (AUC, ROC-AUC)   which appears to do pretty much the same thing.  My question is: is either one of these inherently "better" than the other in some form? I also could just use all three. Is this advisable?