I have built an Logistic Regression model in R. The class that I want to predict, is very unbalanced (99 vs 1).   My first finding is that this Logistic model does a better job if I train it on a balanced (50 - 50) train set, instead of on the whole train set (unbalanced (99 - 1). This is, seen the sources on the internet, a common way to deal with unbalanced data ().  But I have doubts about the next two questions:   First Question: To decide my models performance, I used a confusion matrix. I played around with the threshold (when a prediction is classified as 1, or as 0). See this code:   predictions &lt;- predict(mylogit, test_set, type = "response") confusionMatrix(data = as.numeric(predictions &gt; 0.5), test_set$target) # Here I played around with the 0.5 cutoff, to decide when  # my model is performing best on my test set.    So I tweaked the 0.5 until I had the best confusion matrix score on my test set. Is this valid?   In addition,Second question: If my results (in terms of prediction - Confusion matrix) are bad, can I still use the model to see the influence of the factors (coefficients) on the target? So only for describing the data, and not for predicting? And why?
5301
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.  , the proportion of correct classifications among all classifications, is very simple and very "intuitive" measure, yet it may be a . Why does our intuition misguide us here and are there any other problems with this measure?
107903
Say I fit a logistic regression model on training data and test it on test data.  How do I measure accuracy?   Using the "treshold method" where we predict $y = 1$ if the predicted probability is above 0.5, and vice versa, is a very poor method, obviously. I think it would only work well for simple problems but not for complicated data. A second method I know is to calculate a $\sum_{i}$ where each term is either $\log p_i$ if the actual value at the point $i$ is $1$, or $\log(1 - p_i)$ if the actual value is $0$. But testing this on my data, I do not get sensible results. This method seems to be biased in preference of models that predict $p_i \approx 50 \%$ rather than predict extreme probabilities....   What is a simple and robust method?
74770
I applied  two different classifiers against the same validation set. It turns out that classifier A is better than classifier B in terms of ROC curve. However, classifier B is better than classifier A in terms of confusion matrix. How to explain this kind of contradiction?
37325
I am interested in finding out how confident my model (say Logistic Regression) is in predicting the label of a new data point. For example, if it is not confident, I better abstain from making a prediction.   Logistic Regression outputs probabilities, which gives you a notion that there is confidence in prediction. But in fact, it is not. $P(y|x)$ being 0.3, doesn't tell me that the model is confident in its prediction or not. All, we can say is that it believes 100% that class A is 0.3 and other class is 0.7.  Can we use confidence intervals of LR as some sort of confidence in prediction? E.g., larger the difference between upper and lower bound for a particular data point, less confident it is, and vice-versa?  If not, how can we build confidence in our prediction? Can anybody guide me to some paper or field of study?
22252
I'll explain my problem with an example.  Suppose you want to predict the income of an individual given some attributes: {Age, Gender, Country, Region, City}. You have a training dataset like so  train &lt;- data.frame(CountryID=c(1,1,1,1, 2,2,2,2, 3,3,3,3),               RegionID=c(1,1,1,2, 3,3,4,4, 5,5,5,5),               CityID=c(1,1,2,3, 4,5,6,6, 7,7,7,8),               Age=c(23,48,62,63, 25,41,45,19, 37,41,31,50),               Gender=factor(c("M","F","M","F", "M","F","M","F", "F","F","F","M")),              Income=c(31,42,71,65, 50,51,101,38, 47,50,55,23)) train    CountryID RegionID CityID Age Gender Income 1          1        1      1  23      M     31 2          1        1      1  48      F     42 3          1        1      2  62      M     71 4          1        2      3  63      F     65 5          2        3      4  25      M     50 6          2        3      5  41      F     51 7          2        4      6  45      M    101 8          2        4      6  19      F     38 9          3        5      7  37      F     47 10         3        5      7  41      F     50 11         3        5      7  31      F     55 12         3        5      8  50      M     23   Now suppose I want to predict the income of a new person who lives in City 7.  My training set has a whopping 3 samples with people in City 7 (assume this is a lot) so I can probably use the average income in City 7 to predict the income of this new individual.  Now suppose I want to predict the income of a new person who lives in City 2. My training set only has 1 sample with City 2 so the average income in City 2 probably isn't a reliable predictor.  But I can probably use the average income in Region 1.  Extrapolating this idea a bit, I can transform my training dataset as      Age Gender CountrySamples CountryIncome RegionSamples RegionIncome CitySamples CityIncome  1:  23      M              4         52.25             3        48.00           2    36.5000  2:  48      F              4         52.25             3        48.00           2    36.5000  3:  62      M              4         52.25             3        48.00           1    71.0000  4:  63      F              4         52.25             1        65.00           1    65.0000  5:  25      M              4         60.00             2        50.50           1    50.0000  6:  41      F              4         60.00             2        50.50           1    51.0000  7:  45      M              4         60.00             2        69.50           2    69.5000  8:  19      F              4         60.00             2        69.50           2    69.5000  9:  37      F              4         43.75             4        43.75           3    50.6667 10:  41      F              4         43.75             4        43.75           3    50.6667 11:  31      F              4         43.75             4        43.75           3    50.6667 12:  50      M              4         43.75             4        43.75           1    23.0000   So, the goal is to somehow combine the average CityIncome, RegionIncome, and CountryIncome while using the number of training samples for each to give a weight/credibility to each value. (Ideally, still including information from Age and Gender.)  What are tips for solving this type of problem? I prefer to use tree based models like random forest or gradient boosting, but I'm having trouble getting these to perform well.  UPDATE  For anyone willing to take a stab at this problem, I've generated sample data to test your proposed solution .
103534
I'm trying to solve  from the UCI database repository. Unfortunately (or fortunately), I've noticed that my dataset is imbalanced. I've structured the data as 5 classes, according to the final mark reached by the pupil, like so:   If student gets a mark from 0 to 7 => class 1 [FAIL(E)] If student gets a mark from 8 to 9 => class 2 [SUFFICIENT(D)] If student gets a mark from 10 to 11 => class 3 [GOOD(C)] If student gets a mark from 12 to 15 => class 4 [NOTABLE(B)] If student gets a mark from 16 to 19 => class 5 [OUTSTANDING(A)]   My problem is that, as I said, data are imbalanced, so I want to balance it.  I've thought about applying some kind of undersampling method, but my dataset has only 649 instances so I think removing some of them is not the best idea. Then I thought about doing some oversampling, in order to replicate some of the minority class examples and then get classes balanced, but I'm still unsure if that could work.   I would be very grateful if you could give me a hand with this. Its the first time I face a real problem with imbalanced data.
122094
I have 100,000 observations (9 dummy indicator variables) with 1000 positives.  Logistic Regression should work fine in this case but the cutoff probability puzzles me.    In common literature, we choose 50% cutoff to predict 1s and 0s.  I cannot do this as my model gives a maximum value of ~1%.  So a threshold can be at 0.007 or somewhere around it.    I do understand ROC curves and how the area under curve can help me choose between two LR models for the same dataset.  However, ROC doesn't help me choose an optimum cutoff probability that can be used to test the model on an out-of-sample data.   Should I simply use a cutoff value that minimizes the misclassification rate? ()  Added --> For such a low event rate, my misclassificiation rates are affected by a huge number of false positives.  While the rate over all appears good as total universe size is also big, but my model should not have so many false positives (as it is an investment return model).  5/10 coeff are significant.
6491
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.   My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).   But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?
90700
Is it possible to overfit a logistic regression model? I saw a video saying  that if my area  under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to overfit a logistic regression model?
87318
I have created a Logistic Regression using the following code:  full.model.f = lm(Ft_45 ~ ., LOG_D) base.model.f = lm(Ft_45 ~ IP_util_E2pl_m02_flg) step(base.model.f, scope=list(upper=full.model.f, lower=~1),      direction="forward", trace=FALSE)   I have then used the output to create a final model:  final.model.f = lm(Ft_45 ~ IP_util_E2pl_m02_flg + IP_util_E2_m02_flg +                             AE_NumVisit1_flg + OP_NumVisit1_m01_flg + IP_TotLoS_m02 +                             Ft1_45 + IP_util_E1_m05_flg + IP_TotPrNonElecLoS_m02 +                             IP_util_E2pl_m03_flg + LTC_coding + OP_NumVisit0105_m03_flg +                            OP_NumVisit11pl_m03_flg + AE_ArrAmb_m02_flg)   Then I have predicted the outcomes for a different set of data using the predict function:  log.pred.f.v &lt;- predict(final.model.f, newdata=LOG_V)   I have been able to use establish a pleasing ROC curve and created a table to establish the sensitivity and specificity which gives me responses I would expect.   However What I am trying to do is establish for each row of data what the probability is of Ft_45 being 1. If I look at the output of log.pred.f.v I get, for example,:  1 -0.171739593     2 -0.049905948     3 0.141146419     4 0.11615669     5 0.07342591     6 0.093054334     7 0.957164383     8 0.098415639     . . . 104 0.196368229     105 1.045208447     106 1.05499112   As I only have a tentative grasp on what I am doing I am struggling to understand how to interpret the negative and higher that 1 values as I would expect a probability to be between 0 and 1.  So my question is am I just missing a step where I need to transform the output or have I gone completely wrong. Thank you in advance for any help you are able to offer.
55264
So I run logistic regressions of same dependent variables and independent variables on two sets of data $S_1$ and $S_2$. I obtained two sets of coefficients $\beta_1$ and $\beta_2$. Now I suspect that $\beta_1$ and $\beta_2$ should really be equal, since I don't think there are distinctive difference between $S_1$ and $S_2$.   My question is, how do I test the significant differences between the coefficients?   Thanks!
21830
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .  def FPR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fp / (fp+tn)     return result def FNR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fn / (tp+fn)     return result FPR_scorer = make_scorer(FPR) FNR_scorer = make_scorer(FNR)     def get_CrossValResults2(model,cv_rst,bestIndx):     best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]     roc=&quot;{:.12f}&quot;.format(best['mean_test_roc_auc'])     acc =&quot;{:.0%}&quot;.format(best['mean_test_accuracy'])     prec =&quot;{:.0%}&quot;.format(best['mean_test_precision'])     rec =&quot;{:.0%}&quot;.format( best['mean_test_recall'])     f1 =&quot;{:.0%}&quot;.format(best['mean_test_f1'])     r2=&quot;{:.2f}&quot;.format(best['mean_test_r2'])     g_mean=&quot;{:.2f}&quot;.format(best['mean_test_gmean'])     pr_auc=&quot;{:.8f}&quot;.format(best['mean_test_pr'])     fnr=&quot;{:.0%}&quot;.format(best['mean_test_fnr'])     fpr=&quot;{:.0%}&quot;.format(best['mean_test_fpr'])     rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])     return rst        cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])              scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}     param_grid = {'n_estimators': [200],               'max_depth': [80,90],               'min_samples_leaf': [2,3, 4],             'min_samples_split': [2,5,12],             'criterion': [ 'gini'],              'class_weight' : [class_weights], 'n_jobs' : [-1]}  clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model bestmodel = clf.fit(X,Y)       cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)
62405
I want to perform logistic regression with the following binomial response and with $X_1$ and $X_2$ as my predictors.      I can present the same data as Bernoulli responses in the following format.    The logistic regression outputs for these 2 data sets are mostly the same.  The deviance residuals and AIC are different.  (The difference between the null deviance and the residual deviance is the same in both cases - 0.228.)  The following are the regression outputs from R. The data sets are called binom.data and bern.data.  Here is the binomial output.  Call: glm(formula = cbind(Successes, Trials - Successes) ~ X1 + X2,      family = binomial, data = binom.data)  Deviance Residuals:  [1]  0  0  0  Coefficients:             Estimate Std. Error z value Pr(&gt;|z|) (Intercept)  -2.9649    21.6072  -0.137    0.891 X1Yes        -0.1897     2.5290  -0.075    0.940 X2            0.3596     1.9094   0.188    0.851  (Dispersion parameter for binomial family taken to be 1)  Null deviance:  2.2846e-01  on 2  degrees of freedom Residual deviance: -4.9328e-32  on 0  degrees of freedom AIC: 11.473  Number of Fisher Scoring iterations: 4   Here is the Bernoulli output.  Call: glm(formula = Success ~ X1 + X2, family = binomial, data = bern.data)  Deviance Residuals:      Min       1Q   Median       3Q      Max   -1.6651  -1.3537   0.7585   0.9281   1.0108    Coefficients:             Estimate Std. Error z value Pr(&gt;|z|) (Intercept)  -2.9649    21.6072  -0.137    0.891 X1Yes        -0.1897     2.5290  -0.075    0.940 X2            0.3596     1.9094   0.188    0.851  (Dispersion parameter for binomial family taken to be 1)  Null deviance: 15.276  on 11  degrees of freedom Residual deviance: 15.048  on  9  degrees of freedom AIC: 21.048  Number of Fisher Scoring iterations: 4   My questions:   I can see that the point estimates and standard errors between the 2 approaches are equivalent in this particular case.  Is this equivalence true in general? How can the answer for Question #1 be justified mathematically?   Why are the deviance residuals and AIC different?
118926
I'm doing a project in predicting default risk for SBA loans using R. My data has 187,986 positives and only 17796 negatives (9.46%). Clearly this is an imbalanced dataset. That's why when I run accuracy rate checking for my model using probability cutoff value=0.5, the accuracy result is way too low as below,  &gt; mytestset$LoanStatus_PIF&lt;-ifelse(mytestset$LoanStatus_PIF=="PIF",1,0) &gt; pred&lt;-predict(final,newdata=mytestset,type="response") &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF)   Result: [1] 0.009682553  When I change the cutoff value to 0.998 and up, the result is way much impressive:  &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF) Result:  [1] 0.9997752   Question: 1. Will my model be rejected because I manually choose the cut-off value?   Does 0.999 make sense for a cutoff value? As common cut-off value is usually 0.5. Thanks !
109245
I've got some data about airline flights (in a data frame called flights) and I would like to see if the flight time has any effect on the probability of a significantly delayed arrival (meaning 10 or more minutes). I figured I'd use logistic regression, with the flight time as the predictor and whether or not each flight was significantly delayed (a bunch of Bernoullis) as the response. I used the following code...  flights$BigDelay &lt;- flights$ArrDelay &gt;= 10 delay.model &lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link="logit")) summary(delay.model)   ...but got the following output.  &gt; flights$BigDelay &lt;- flights$ArrDelay &gt;= 10 &gt; delay.model &lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link="logit")) Warning messages: 1: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :   algorithm did not converge 2: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :   fitted probabilities numerically 0 or 1 occurred &gt; summary(delay.model)  Call: glm(formula = BigDelay ~ ArrDelay, family = binomial(link = "logit"),     data = flights)  Deviance Residuals:        Min          1Q      Median          3Q         Max -3.843e-04  -2.107e-08  -2.107e-08   2.107e-08   3.814e-04  Coefficients:             Estimate Std. Error z value Pr(&gt;|z|) (Intercept)  -312.14     170.26  -1.833   0.0668 . ArrDelay       32.86      17.92   1.833   0.0668 . --- Signif. codes:  0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1  (Dispersion parameter for binomial family taken to be 1)      Null deviance: 2.8375e+06  on 2291292  degrees of freedom Residual deviance: 9.1675e-03  on 2291291  degrees of freedom AIC: 4.0092  Number of Fisher Scoring iterations: 25   What does it mean that the algorithm did not converge? I thought it be because the BigDelay values were TRUE and FALSE instead of 0 and 1, but I got the same error after I converted everything. Any ideas?
38281
Why is it that logistic regression becomes unstable when classes are well-separated? What does well-separated classes mean? I would really appreciate if someone can explain with an example.
93095
I've been thinking a lot about the &quot;class imbalance problem&quot; in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on. First let me define (or attempt to) define my terms:  The class imbalance problem in machine/statistical learning is the observation that some binary classification(*) algorithms do not perform well when the proportion of 0 classes to 1 classes is very skewed.  So, in the above, for example, if there were one-hundred $0$ classes for every single $1$ class, I would say the class imbalance is $1$ to $100$, or $1\%$. Most statements of the problem I have seen lack what I would think of as sufficient qualification (what models struggle, how imbalanced is a problem), and this is one source of my confusion. A survey of the standard texts in machine/statistical learning turns up little:  Elements of Statistical Leaning and Introduction to Statistical Learning do not contain &quot;class imbalance&quot; in the index.  Machine Learning for Predictive Data Analytics also does not contain&quot;class imbalance&quot; in the index.  Murphy's Machine Learning: A Probabilistic Perspective does contain &quot;class imbalance* in the index.  The reference is to a section on SVM's, where I found the following tantalizing comment:  It is worth remembering that all these difficulties, and the plethora of heuristics that have been proposed to fix them, fundamentally arise because SVM's do not model uncertainty using probabilities, so their output scores are not comparable across classes.    This comment does jive with my intuition and experience: at my previous job we would routinely fit logistic regressions and gradient boosted tree models (to minimize binomial log-likelihood) to unbalanced data (on the order of a $1\%$ class imbalance), with no obvious issues in performance. I have read (somewhere) that classification tree based models (trees themselves and random forest) do also suffer from the class imbalance problem.  This muddies the waters a little bit, trees do, in some sense, return probabilities: the voting record for the target class in each terminal node of the tree. So, to wrap up, what I'm really after is a conceptual understanding of the forces that lead to the class imbalance problem (if it exists).  Is it something we do to ourselves with badly chosen algorithms and lazy default classification thresholds? Does it vanish if we always fit probability models that optimize proper scoring criteria?  Said differently, is the cause simply a poor choice of loss function, i.e. evaluating the predictive power of a model based on hard classification rules and overall accuracy? If so, are models that do not optimize proper scoring rules then useless (or at least less useful)?  (*) By classification I mean any statistical model fit to binary response data.  I am not assuming that my goal is a hard assignment to one class or the other, though it may be.