This question relates to whether it is a good starting point for a cut point in binary classification with logistic regression to the use the mean of the binary response variable as the initial cut point rather than simply 0.5.  Traditionally when people use logistic regression, people with use 0.5 as the threshold to determine when the model predicts YES/positive versus NO/negative.  People may run into trouble when the model only predicts one "answer" when using an imbalanced training set.  One way of dealing with this is to balance the training set via oversampling or under-sampling and keeping the test holdout set with the original balance.  However, I suspect that a good starting point for a cut point appears to be the mean of the binary response variable.  Is this usually true?  I created two models, one on a balanced training set and another on the original imbalanced training set. print(table(actual=test$y, predicted=test$fit&gt;0.5))         predicted  actual FALSE TRUE       0  2359  500       1    11  130   With the imbalanced training, I used the mean of the binary response variable:  print(table(actual=test$y, predicted=test$fit&gt;0.0496))         predicted  actual FALSE TRUE       0  2317  542       1     7  134   If one just uses 0.5, it looks like the model is a complete failure:  `print(table(actual=test$y, predicted=test$fit&gt;0.5))`         predicted  actual FALSE       0  2848       1   152   They both had a KS of 0.76, so it seems like sound advice.  Example R code:  require(ROCR) require(lattice) # x=1:10000/10000; y=ifelse(runif(10000)-0.7&gt;jitter(x),1,0) #y=ifelse(rnorm(10000)-0.99&gt;x,1,0) mean(y)  s=sample(length(x),length(x)*0.7);  df=data.frame(x=x,y=y)   ##undersample train=df[s,] train=rbind(train[train$y==1,],train[sample(which(train$y==0),sum(train$y==1)),]) ##oversample train=df[s,] train=rbind(train[train$y==0,],train[sample(which(train$y==1),sum(train$y==0),replace = T),]) mean(train$y) #now balanced threshold=0.5 test=df[-s,] #unbalanced mean(test$y) #  ex=glm(y~x,train, family = "binomial") summary(ex) nrow(test) test$fit=predict(ex,newdata = test,type="response") message("threshold=",threshold) print(table(actual=test$y, predicted=test$fit&gt;threshold))   #+results pred&lt;-prediction(test$fit,test$y) perf &lt;- performance(pred,"tpr","fpr") ks.sc=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]]) plot(perf) print(ks.sc); #ks.score   levelplot(fit~y+x,test,col.regions =  terrain.colors(100)[1:95])   #+ imbalanced approach #############imbalance approach  train=df[s,] threshold=mean(y) message("threshold=",threshold) ex=glm(y~x,train, family = "binomial") summary(ex) test$fit=predict(ex,test,type = "response") summary(test$fit) print(table(actual=test$y, predicted=test$fit&gt;threshold))   print(table(actual=test$y, predicted=test$fit&gt;0.5))   pred&lt;-prediction(test$fit,test$y) perf &lt;- performance(pred,"tpr","fpr") ks.sc=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]]) plot(perf) print(ks.sc); #ks.score   levelplot(fit~y+x,test,col.regions =  terrain.colors(100)[1:95])    I noticed a similar question asked   I like the answer given here which states to maximize the specificity or sensitivity:   But I also suspect that the usual starting cut off of 0.5 is bad advice.  Comments?
122094
I have 100,000 observations (9 dummy indicator variables) with 1000 positives.  Logistic Regression should work fine in this case but the cutoff probability puzzles me.    In common literature, we choose 50% cutoff to predict 1s and 0s.  I cannot do this as my model gives a maximum value of ~1%.  So a threshold can be at 0.007 or somewhere around it.    I do understand ROC curves and how the area under curve can help me choose between two LR models for the same dataset.  However, ROC doesn't help me choose an optimum cutoff probability that can be used to test the model on an out-of-sample data.   Should I simply use a cutoff value that minimizes the misclassification rate? ()  Added --> For such a low event rate, my misclassificiation rates are affected by a huge number of false positives.  While the rate over all appears good as total universe size is also big, but my model should not have so many false positives (as it is an investment return model).  5/10 coeff are significant.
48982
I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than imbalance in training data.  In other words, only changing the distribution of positive and negative classes in the test data, the AUC value may not change much. But if we change the distribution in the training data, the AUC value may largely change. The reason is that the classifier cannot be learned well. In this case, we have to use undersampling and oversampling. Am I right? I just want to make sure my understanding on AUC is correct.
104639
I searched on topics of the bias and variance trade-off and got back lots of questions with different levels of response. The information is scattering too much and unsystematic to answer my own question which stuck me. So I open a new question here and would like to hackle how I generally run a machine learning modelling process and ask for help on the issue of bias-variance assessment where I was confused or had no idea. If anyone noticed any useful posts to my questions please help point out the links and many thanks.  I execute the process basically according to the instruction below:    Question 1: When I start at step one, I try to think an issue that all my following analysis will be based on this train-test split. How about another split and is it possible to cause an obvious change in analysis results (final models, performance metrics, important features etc.)? If so, that means the analysis was split-dependent.   I have some thoughts here: assuming there exists an unknown underlying distribution of the domain set X (the whole dataset except the target class y), if the sample distribution of training set S was approximately similar to that of test set T, then the learning process should reach a good performance. Otherwise, if the two sample distributions differed in some non-trivial way, then the learner/classifier from training set S may not probably do a good job on test set T.     I made a trial to execute a 10-fold cross validation on the whole dataset and to see how variable the ROC AUC score was. In my particular dataset, I got a set of 10 scores with large variation (from maximum 0.85 to minimum 0.59). This led to my worry whether the current split I adopted for analysis was poor one.  To expand Question 1, I would like to ask further: what is the topic of this issue in machine learning theory? Is it the model stability or model variance?  I tuned the hyper-parameter/parameters in step 2 via cross validation. I try to use validation curve (refer to ) as below to choose the parameter value and avoid under/over-fitting problems.     Question 2: If this curve worked well in detecting under/over-fitting problems, what metric should be used: accuracy score/error rate, presicion, recall, f1 or roc_auc? (My current project was a classification problem with an imbalanced binary target class.)  According to the definition of bias, average error rate could be the measure of bias, so accuracy score/error rate seemed more reasonable in bias/variance issue. In fact, I concerned roc_auc score more than accuracy itself.   Question 3: how can we measure the variance? I saw a plot  and the gap was said to be able to indicate the potential variance (poorer performance on validation set than that on training set). Any other (better) way/metric can be used for this point?    In my analysis, the bias/variance check was run in the parameter tuning stage which seemed to return me a candidate model with a balanced bias/variance combination.  Question 4: does my bias/variance analysis miss some steps and how do I adjust it?  many ensemble learning methods have their own way to reduce variance or bias and  I hope to objectively evaluate the bias/variance for each candidate model.   Thanks for reading my long question but it really help for me to deliver a professional practice in my daily work. Many thanks again.
104507
Problem statement: Evaluate a binary classifier. There are 50 positive outcomes in the test data, and 100 observations. Using a 50% threshold, the classifier predicts 40 positive outcomes, of which 10 are incorrect. Calculate classifier’s Precision and Recall on the test sample? My tried solution: I have calculated Precision as : P = TP / TP + FP =&gt; P = 40 / 40 + 10 =&gt; P = 0.8 And, same Recall. But, I am not sure its correct as F1 score options are not matching with mine. So, I need to build a correct 2x2 Confusion matrix. Please let me know for any query or information required to solve this question. Thanks
19394
I am building a classification model (let's say logistic regression) using 2 different data sets. The data sets do not come from the same population. Let's say the data sets comes from different space-time coordinates (Asia in Fall vs North America in Winter). I have been told that simply mixing the data sets into one and applying a classification model is not right (closest thing I understood was concerns around bias). Are there any techniques or models that I may use to analyze these 2 data sets as one? Note that the variables/features all align (including the target variable, which is binary).
55346
I need help thinking about and identifying the kind of regression analysis that would be appropriate for this problem. Nothing I've discovered so far seems quite right. Referrals to articles or examples would be helpful. Thank you.  The data look like this:   The data are observational. The sampling unit is a geographic location (EDIT: let's assume units are independent); I'm just trying to understand the basic analytical problem here). At each sampling unit, there are events of two types: the event type of interest (A) and all other types (B). EDIT: In other words, each event is a binary outcome (success, failure). The outcomes are aggregated to the to the location level (Location 1: Success 3, Failure 2. Location 2: Success 0, Failure 1. Location 3: Success 0, Failure 0. Location 4: Success 4, Failure 9 .... etc.  Often, A=0 and, somewhat less often but still frequently, A+B=0.    I am interested in testing a hypothesis about A, somehow controlling for the total count of events (A+B), so either a proportion A/(A+B) or a count model that controls for the total count.  If I understand correctly, if the counts were large, proportions could be calculated for all units, and I could do beta regression. But that definitely can't happen when the total number of events for a unit is zero.  If all I cared about was the count, I could use a ZIP or other count model (and maybe still can). But the research question regards the frequency of A relative to the total number of events.  But how to control for the total number of events? Does it just go in the predictors of a ZIP or similar model? I suspect it's more complicated than that.  It seems obvious to me that the individual events could be modeled directly using multi-level logistic regression (EDIT: or another model for clustered data), but I'm wondering if there is a simpler way to examine what I'm interested in, and I just somehow haven't seen an example of this.
109245
I've got some data about airline flights (in a data frame called flights) and I would like to see if the flight time has any effect on the probability of a significantly delayed arrival (meaning 10 or more minutes). I figured I'd use logistic regression, with the flight time as the predictor and whether or not each flight was significantly delayed (a bunch of Bernoullis) as the response. I used the following code...  flights$BigDelay &lt;- flights$ArrDelay &gt;= 10 delay.model &lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link="logit")) summary(delay.model)   ...but got the following output.  &gt; flights$BigDelay &lt;- flights$ArrDelay &gt;= 10 &gt; delay.model &lt;- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link="logit")) Warning messages: 1: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :   algorithm did not converge 2: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :   fitted probabilities numerically 0 or 1 occurred &gt; summary(delay.model)  Call: glm(formula = BigDelay ~ ArrDelay, family = binomial(link = "logit"),     data = flights)  Deviance Residuals:        Min          1Q      Median          3Q         Max -3.843e-04  -2.107e-08  -2.107e-08   2.107e-08   3.814e-04  Coefficients:             Estimate Std. Error z value Pr(&gt;|z|) (Intercept)  -312.14     170.26  -1.833   0.0668 . ArrDelay       32.86      17.92   1.833   0.0668 . --- Signif. codes:  0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1  (Dispersion parameter for binomial family taken to be 1)      Null deviance: 2.8375e+06  on 2291292  degrees of freedom Residual deviance: 9.1675e-03  on 2291291  degrees of freedom AIC: 4.0092  Number of Fisher Scoring iterations: 25   What does it mean that the algorithm did not converge? I thought it be because the BigDelay values were TRUE and FALSE instead of 0 and 1, but I got the same error after I converted everything. Any ideas?
1463
Let's say I am building a logistic regression model where the dependent variable is binary and can take the values $0$ or $1$. Let the independent variables be $x_1, x_2, ..., x_m$ - there are $m$ independent variables. Let's say for the $k$th independent variable, the bivariate analysis shows a U-shaped trend - i.e., if I group $x_k$ into $20$ bins each containing roughly equal number of observations and calculate the 'bad rate' for each bin - # observations where y = 0 / total observations in each bin - then I get a U shaped curve.  My questions are:   Can I directly use $x_k$ as input while estimating the beta parameters? Are any statistical assumptions violated which might cause significant error in estimating the parameters? Is it necessary to 'linearize' this variable through a transformation (log, square, product with itself, etc.)?
60604
First I'm a little bit confused with the assumptions of a model with binomial response and logit link function. Since it is basically a logistic regression. Today one of my professors told me that I had to prove normality of my residuals, but of what I know the assumptions are:  First, binary logistic regression requires the dependent variable to be binary and ordinal logistic regression requires the dependent variable to be ordinal.  Second, logistic regression requires the observations to be independent of each other.  In other words, the observations should not come from repeated measurements or matched data.  Third, logistic regression requires there to be little or no multicollinearity among the independent variables.  This means that the independent variables should not be too highly correlated with each other.  Also these are the plots I have:     So basically my question is: what assumptions can I prove/not prove with these plots?
87872
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.  My question is, are there any rules of thumb that tell us when we should subsample the large category in order to force some kind of balancing in the dataset.  Examples:   If the number of positive examples is 1,000 and the number of negative examples is 10,000, should I go for training my classifier on the full dataset or I should subsample the negative examples? The same question for 1,000 positive example and 100,000 negative. The same question for 10,000 positive and 1,000 negative. etc...
15933
I've been thinking about this problem.  The usual logistic function for modeling binary data is: $$ \log\left(\frac{p}{1-p}\right)=\beta_0+\beta_1X_1+\beta_2X_2+\ldots $$ However is the logit function, which is an S-shaped curve, always the best for modeling the data?  Maybe you have reason to believe your data does not follow the normal S-shaped curve but a different type of curve with domain $(0,1)$.  Is there any research into this?  Maybe you can model it as a probit function or something similar, but what if it is something else entirely?  Could this lead to better estimation of the effects?  Just a thought I had, and I wonder if there is any research into this.
29978
This is somewhat of a beginner's question, but how does one interpret an exp(B) result of 6.012 in a multinomial logistic regression model?  1) is it 6.012-1.0 = 5.012 = 5012% increase in risk?  or  2) 6.012/(1+6.012) = 0.857 = 85.7% increase in risk?  In case both alternatives are incorrect, can someone please mention the correct way?  Ive searched many resources on the internet and I get to these two alternatives, and I am not entirely sure which one is correct.
92944
I frequently hear up- or down-sampling of data discussed as a way of dealing with classification of imbalanced data.  I understand that this could be useful if you're working with a binary (as opposed to probabilistic or score-based) classifier and treating it as a black box, so sampling schemes are your only way to tweak its position on the "ROC curve" (in quotes because if your classifier is inherently binary I guess it doesn't have a real ROC curve, but the same concept of trading off false positives and false negatives still applies).  But it seems like the same justification doesn't hold if you actually have access to some kind of score that you're later thresholding to make a decision. In this case, isn't up-sampling just an ad-hoc way of expressing a view about your desired trade-off between false positives and false negatives when you have much better tools available, like actual ROC analysis? It seems like it would be weird in this case to expect up-sampling or down-sampling to do anything but change your classifier's "prior" on each class (i.e. unconditional probability of being that class, the baseline prediction)--I wouldn't expect it to change the classifier's "odds ratio" (how much the classifier adjusts its baseline prediction based on the covariates).  So my question is: if you have a classifier that isn't a binary black box, are there any reasons to expect up- or down-sampling to have a much better effect than adjusting the threshold to your liking? Failing that, are there any empirical studies showing reasonably large effects for up- or down-sampling on reasonable performance metrics (e.g., not accuracy)?
29229
I'm modeling (using GLM) one binary explanatory variable against a binary response variable.  The output looks ok to me, but I'm a biologist with a basic knowledge of stats, I just wanted to ask if there's any rule/caveat/problem for modeling a binary versus a binary?
17183
I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model.  For example, in the validation dataset, I have the true value for the dependent variable, retention (1 = retained; 0 = not retained), as well as a predicted retention status for each observation generated by my regression analysis using a model that was built using the training set (this will range from 0 to 1).  My initial thoughts were to identify the "correct" number of model classifications and simply divide the number of "correct" observations by the number of total observations to calculate the c-statistic. By "correct", if the true retention status of an observation = 1 and the predicted retention status is > 0.5 then that is a "correct" classification. Additionally, if the true retention status of an observation = 0 and the predicted retention status is &lt; 0.5 then that is also a "correct" classification. I assume a "tie" would occur when the predicted value = 0.5, but that phenomenon does not occur in my validation dataset. On the other hand, "incorrect" classifications would be if the true retention status of an observation = 1 and the predicted retention status is &lt; 0.5 or if the true retention status for an outcome = 0 and the predicted retention status is > 0.5. I am aware of TP, FP, FN, TN, but not aware of how to calculate the c-statistic given this information.
104452
I'm looking for optimal binning method (discretization) of a continuous variable with respect to a given response (target) binary variable and with maximum number of intervals as a parameter.  example: I have a set of observations of people with "height" (numeral continuous) and "has_back_pains" (binary) variables. I want do discretize height into 3 intervals (groups) at most with different proportion of people with back aches, so that the algorithm maximizes the difference between the groups (with given restrictions for instance, that each interval has at least x observations).  The obvious solution to this problem would be to use decision trees(a simple one-variable model) , but I can't find any function in R that would have "maximal number of branches" as a parameter - all of them divide the variable into 2 gropus  (&lt;=x and >x). SAS miner has a "maximum branch" parameter but I'm looking for a non commercial solution.  some of my variables have just a few unique values (and could be treated as discrete variables) but I want to discretize them as well into a smaller number of intervals.  The closest solution to my problem is implemented in the smbinning package in R (which relies on ctree function from party package) but it has two drawbacks: it's impossible to set the number of intervals (however, you can find a way around it by changing the p parameter) and it doesn't work when data vector has less than 10 unique values. Anyway, you can see the example output here(Cutpoint and Odds columns are crucial):  Cutpoint CntRec CntGood CntBad CntCumRec CntCumGood CntCumBad PctRec BadRate   Odds  LnOdds     WoE     IV 1   &lt;= 272   9081     169   8912      9081        169      8912 0.1874  0.9814 0.0190 -3.9653 -0.6527 0.0596 2   &lt;= 311   8541     246   8295     17622        415     17207 0.1762  0.9712 0.0297 -3.5181 -0.2055 0.0068 3   &lt;= 335   2986     163   2823     20608        578     20030 0.0616  0.9454 0.0577 -2.8518  0.4608 0.0163 4  Missing  27852    1125  26727     48460       1703     46757 0.5747  0.9596 0.0421 -3.1679  0.1447 0.0129 5    Total  48460    1703  46757        NA         NA        NA 1.0000  0.9649 0.0364 -3.3126  0.0000 0.0956   Oh, I'm fully aware that binning results in information loss and that there are better methods, but I'm going to use it for data visualization and treat those variables as a factor.
6137
I am thinking of building a model predicting a ratio $a/b$, where $a \le b$ and $a &gt; 0$ and $b &gt; 0$. So, the ratio would be between $0$ and $1$.  I could use linear regression, although it doesn't naturally limit to 0..1.  I have no reason to believe the relationship is linear, but of course it is often used anyway, as a simple first model.  I could use a logistic regression, although it is normally used to predict the probability of a two-state outcome, not to predict a continuous value from the range 0..1.  Knowing nothing more, would you use linear regression, logistic regression, or hidden option c?