I am having the transaction dataset which contains 95% or user initiated transactions and 5% fraud transactions. When I am fitting the logistic regression model, it gives me bias prediction - as most of the transactions are as user initiated. So, I want to fix this issue of data distribution or other issue?  One of source suggested that prior probability can help in this issue. If this is true can any one explain? What's it and if this is not helpful then which are the possible solution for this problem?  I have other user profile variables but I want to get this thing solved without considering them. Is there any solution ?
6491
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.   My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).   But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?
116146
From this , the following statement is posed: 'Though not "wrong", you'd want a good reason for using an identity link to model a Bernoulli probability.'  I would like to know what good reasons would result in an identity link being used for logistic regression, if these exist. Do these reasons generalise to GLMs?
17662
If I have a dataset with a very rare positive class, and I down-sample the negative class, then perform a logistic regression, do I need to adjust the regression coefficients to reflect the fact that I changed the prevalence of the positive class?  For example, let's say I have a dataset with 4 variables: Y, A, B and C.  Y, A, and B are binary, C is continuous.  For 11,100 observations Y=0, and for 900 Y=1:  set.seed(42) n &lt;- 12000 r &lt;- 1/12 A &lt;- sample(0:1, n, replace=TRUE) B &lt;- sample(0:1, n, replace=TRUE) C &lt;- rnorm(n) Y &lt;- ifelse(10 * A + 0.5 * B + 5 * C + rnorm(n)/10 &gt; -5, 0, 1)   I fit a logistic regression to predict Y, given A, B and C.   dat1 &lt;- data.frame(Y, A, B, C) mod1 &lt;- glm(Y~., dat1, family=binomial)   However, to save time I could remove 10,200 non-Y observations, giving 900 Y=0, and 900 Y=1:  require('caret') dat2 &lt;- downSample(data.frame(A, B, C), factor(Y), list=FALSE) mod2 &lt;- glm(Class~., dat2, family=binomial)   The regression coefficients from the 2 models look very similar:  &gt; coef(summary(mod1))               Estimate Std. Error   z value     Pr(&gt;|z|) (Intercept) -127.67782  20.619858 -6.191983 5.941186e-10 A           -257.20668  41.650386 -6.175373 6.600728e-10 B            -13.20966   2.231606 -5.919353 3.232109e-09 C           -127.73597  20.630541 -6.191596 5.955818e-10 &gt; coef(summary(mod2))               Estimate  Std. Error     z value    Pr(&gt;|z|) (Intercept) -167.90178   59.126511 -2.83970391 0.004515542 A           -246.59975 4059.733845 -0.06074284 0.951564016 B            -16.93093    5.861286 -2.88860377 0.003869563 C           -170.18735   59.516021 -2.85952165 0.004242805   Which leads me to believe that the down-sampling did not affect the coefficients.  However, this is a single, contrived example, and I'd rather know for sure.
118926
I'm doing a project in predicting default risk for SBA loans using R. My data has 187,986 positives and only 17796 negatives (9.46%). Clearly this is an imbalanced dataset. That's why when I run accuracy rate checking for my model using probability cutoff value=0.5, the accuracy result is way too low as below,  &gt; mytestset$LoanStatus_PIF&lt;-ifelse(mytestset$LoanStatus_PIF=="PIF",1,0) &gt; pred&lt;-predict(final,newdata=mytestset,type="response") &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF)   Result: [1] 0.009682553  When I change the cutoff value to 0.998 and up, the result is way much impressive:  &gt; y_pred_num&lt;-ifelse(pred&gt;0.5,1,0) &gt; mean(y_pred_num==mytestset$LoanStatus_PIF) Result:  [1] 0.9997752   Question: 1. Will my model be rejected because I manually choose the cut-off value?   Does 0.999 make sense for a cutoff value? As common cut-off value is usually 0.5. Thanks !
30940
I have a data set for a predictive model(predicting survival rate with certain acute medical condition on some animals) with 25 predictors where around 30% of the predictors are complete, 3 predictors are sitting at 30%,25% and 20% missing, and the others are missing at around 5% level. Around 50% of my data are complete cases. I'm new to dealing with missing values so I have a couple of questions regarding how to deal with it:   What can I do with the variable with 30% missing assuming it's MAR? Is 30% too high for imputation? What kinds of metrics can I use to make the decision between removal of the predictor, listwise removal, imputation or some other options? How should I deal with a predictor with 20% - 25% missing when I have reason to believe that it's MNAR? I'm thinking of using imputation on the remaining predictors with around 5% missingness. How do I make the decision on which imputation methods to use? Are they chosen on a case by case basis based on the individual predictors? How does one impute categorical values?  How does(should?) the imputation carry out in practice? Should it use complete cases or compute it iteratively somehow?  Does feature selection comes in before or after dealing with missing values? if at all?
38281
Why is it that logistic regression becomes unstable when classes are well-separated? What does well-separated classes mean? I would really appreciate if someone can explain with an example.
62633
I am running an analysis on the probability of loan default using logistic regression and random forests.    When I use logistic regression, the prediction is always all '1' (which means good loan).  I have never seen this before, and do not know where to start in terms of trying to sort out the issue.  There are 22 columns with 600K rows.  When I decrease the # of columns I get the same result with logistic regression.   Why could the logistic regression be so wrong?  **Actual from the data**  0 :   41932  1 :   573426  **Logistic regression output**   prediction for 1 when actually 0: 41932 prediction for 1 when actually 1:573426  A**s you can see, it always predicts a 1**   **Random forests does better:**  actual 0, pred 0 : 38800  actual 1, pred 0 : 27  actual 0, pred 1 : 3132 actual 1, pred 1 : 573399
94380
I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of 14 variables.  The results are completely different in the intercept and the weights. How to explain this?  Here is the results of Spark (LogisticRegressionWithLBFGS) :  model.intercept  :   1.119830027739959 model.weights :  GEST    0.30798496002530473  DILATE  0.28121771009716895  EFFACE  0.01780105068588628  CONSIS -0.22782058111362183  CONTR  -0.8094592237248102  MEMBRAN-1.788173534959893  AGE    -0.05285751197750732  STRAT  -1.6650305527536942  GRAVID  0.38324952943210994  PARIT  -0.9463956993328745  DIAB   0.18151162744507293  TRANSF -0.7413500749909346  GEMEL  1.5953124037323745   Here is the result of R :               Estimate Std. Error z value Pr(&gt;|z|)    (Intercept)  3.0682091  3.3944407   0.904 0.366052     GEST         0.0086545  0.1494487   0.058 0.953821     DILATE       0.4898586  0.2049361   2.390 0.016835 *   EFFACE       0.0131834  0.0059331   2.222 0.026283 *   CONSIS       0.1598426  0.2332670   0.685 0.493196     CONTR        0.0008504  0.5788959   0.001 0.998828     MEMBRAN     -1.5497870  0.4215416  -3.676 0.000236 ***    AGE         -0.0420145  0.0326184  -1.288 0.197725     STRAT       -0.3781365  0.5860476  -0.645 0.518777     GRAVID       0.1866430  0.1522925   1.226 0.220366     PARIT       -0.6493312  0.2357530  -2.754 0.005882 **   DIAB         0.0335458  0.2163165   0.155 0.876760     TRANSF      -0.6239330  0.3396592  -1.837 0.066219 .   GEMEL        2.2767331  1.0995245   2.071 0.038391 *   Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
7160
Suppose the event of interest occurs in approximately $10 \%$ of the cases where the number of cases is around $5,000$. Should you use a penalized logistic regression for this or is regular logistic regression okay? In other words, what qualifies something as a rare event?
54007
How are you dealing with the  in your machine learning models? Let's say you used to predict the number of sales each month. The virus affected your results last year and it will affect for at least a couple of months. So your model, I guess, is missing a lot. What are some approaches we can use to fix our models?
3187
On whether an error term exists in logistic regression (and its assumed distribution), I have read in various places that:   no error term exists the error term has a binomial distribution (in accordance with the distribution of the response variable) the error term has a logistic distribution   Can someone please clarify?
90700
Is it possible to overfit a logistic regression model? I saw a video saying  that if my area  under the ROC curve is higher than 95%, then its very likely to be over fitted, but is it possible to overfit a logistic regression model?
56697
What is your opinion about oversampling in classification in general, and the SMOTE algorithm in particular? Why would we not just apply a cost/penalty to adjust for imbalance in class data and any unbalanced cost of errors? For my purposes, accuracy of prediction to a future set of experimental units is the ultimate measure.  For reference, the SMOTE paper:
26576
I am running a logit regression on a dataset with only 2 independent categorical variables, country and device type. Both of these variables can take several values. The dependant variable is binary. The event occurs approximately 0.2% of the time.  For certain countries, the coefficients are significant, but I do not have any positive event for this country. How can this happen ?  I am using the glm procedure in R.  Thank you,
104358
I am confused with the assumption of linearity to the logit for continuous predictor variables in logistic regression analysis. Do we need to check for the linear relationship while screening for potential predictors using univariable logistic regression analysis?   In my case, I am using the multiple logistic regression analysis to identify factors associated with nutritional status (dichotomous outcome) among the participants. The continuous variables including age, Charlson comorbidity score, Barthel Index score, hand grip strength, GDS score, BMI etc. My first step is to screen for significant variables using simple logistic regression. Do I need to check for the linearity assumption during simple logistic regression analyses for each continuous variables? Or should I just check for it in the final multiple logistic regression model?  Besides, for my understanding, we need to transform the non-linear continuous variable before enter it into the model. Can I categorize the non-linear continuous variable instead of transformation?
17350
I would like to understand a couple of fact on maximum likelihood estimators (MLEs) for logistic regressions.   Is it true that, in general, the MLE for logistic regression is biased? I would say "yes". I know, for example, that sample dimension is related to the asymptotic bias of MLEs.   Do you know any elementary examples of this phenomenon? If the MLE is biased, is it true that the covariance matrix of the MLEs is the inverse of the Hessian of the maximum likelihood function?   edit: I have met this formula quite often and without any proof; it seems a quite arbitrary choice to me.
73535
So, I have been presented with this question: A sample of 100 people were asked how many days they drove their car during the last week (inc. the weekend). The resulting frequency of response is shown below: Days, frequency 0, 1 1, 5 2, 3 3, 15 4, 20 5, 25 6, 31 7, 0 I need to use a beta-binomial model in order to estimate the average number of 'car driven' days a week for a uniform prior. There are a total of 7 Bernoulli trials, each with success probability p. I'm incredibly confused at what the outcome will be. I know that for a uniform prior that alpha = 1, and beta = 1, but I don't know how to apply this all to get the average? Thank you.