As stated in , Gaussian Discriminant Analysis (GDA) can also be expressed in the form of $\frac{1}{1+\exp(-\theta^Tx)}$, where $\theta$ is some appropriate function of $φ$, $\Sigma$, $u_0$, and $u_1$. So why can't logistic regression predict model like them?
33131
I found some pros of discriminant analysis and I've got questions about them. So:     When the classes are well-separated, the parameter estimates for   logistic regression are surprisingly unstable. Coefficients may go to   infinity. LDA doesn't suffer from this problem.      If number of features is small and the distribution of the predictors   $X$ is approximately normal in each of the classes, the linear   discriminant model is again more stable than the logistic regression   model.    What is stability and why is it important? (If logistic regression provides a good fit that does its job, then why should I care about stability?)      LDA is popular when we have more than two response classes, because it   also provides low-dimensional views of the data.    I just don't understand that. How does LDA provide low-dimensional views? If you can name more pros or cons, that would be nice.
105260
Can anybody explain differences and give specific examples how to use these three analyses?   LDA - Linear Discriminant Analysis FDA - Fisher's Discriminant Analysis QDA - Quadratic Discriminant Analysis   I searched everywhere, but couldn't find real examples with real values to see how these analyses are used and data calculated, only lots of formulas which are hard to understand without any real examples. As I tried to understand it was hard to distinguish which equations/formulas belonged to LDA and which to FDA.  For example let's say there is such data:  x1 x2 class 1  2  a 1  3  a 2  3  a 3  3  a 1  0  b 2  1  b 2  2  b   And let's say some testing data:  x1 x2 2  4 3  5 3  6   So how to use such data with all these three approaches? It would be best to see how to calculate everything by hand, not using some math package which calculates everything behind the scenes.   P.S. I only found this tutorial: . It shows how to use LDA.
40303
I am trying to model vegetation spectral signatures (grey lines) using a two-part piecewise function (black dotted lines). In it, I am trying to use only a few points (red dots) to fit a linear (first part) and logistic function (second part). Basically, the linear part of the function stretches a bit further than the second point (how much it stretches, it will depend on &quot;unknown&quot; parameters, but as a rule of thumb I use 35 &quot;X&quot; units). I currently defined an ifelse function and I am applying a constrained optimization (optim, method=&quot;L-BFGS-B&quot;), to find the best parameter values. This has a few handicaps, as the parameters are not normalized/scaled (which makes the search routine less efficient) logistic.fun&lt;- function(K, C, ro, b, Z, a, lambda){       ifelse(test = lambda &lt;= Seq_Bands[2]+25,             yes = a + b*((Seq_Bands[2]+25)- lambda),            no = ((ro*K)/((ro + (K-ro)*exp(-C*(lambda-Seq_Bands[2]+Z))))))}  I would be keen to use nls and fit a 3 parameters logistic regression and also use a fourth parameter to estimate the breakpoint of the linear part (Z parameter code above). Also, I would be keen to avoid jumps such as those seen on left plot below. This means that the functions would have to be differentiable (?) at this breakpoint. I am unsure how to code it. Cheers and thanks!   UPDATE: It was correctly pointed out that the number of variables is higher than the data points presented; which would render the problem under-determined. Consequently, the logistic equation can only be parameterized using 3 variables (Fig 2: R_0_ , K and r. My understanding is that, necessarily, the breakpoint (Z) cannot be estimated, and should be set prior to the optimization process. In context, it is also important to state that the R_0_, and K are not &quot;truly&quot; unknowns, as they are expressed by the measurements (second and fourth red datapoints).
108108
Consider model $logit(p)=a+bx$. I would like to get an analytic formula of $a$ and $b$ like in linear regression. In linear regression, we can get a formula of estimates of $a$ and $b$.  I tried using MLE. But it is too complicated for me.  I am doing research about the statistical genetics. Here $p$ is penetrance and $x$ is genotype. In linear regression, we can use the coefficient $b$ to represent the covariance matrix of $x$ and trait $y$ (both of them should be standardized). I am thinking whether we can use the $b$ in logistic model to recover the information about the covariance matrix of $x$ and $y$ (binary trait) in meta-analysis (they do not provide individual level data). But some cohort will prove the odds ratio and coefficient t-statistic. I do not know how to use it to recover the covariance matrix of $x$ and $y$.   But some summary statistics will be provided like estimated odds ratio, Coefficient t-statistic.
44714
Is there a relationship between regression and linear discriminant analysis (LDA)? What are their similarities and differences? Does it make any difference if there are two classes or more than two classes?
29773
I like Peter Flom's answer to an earlier question about multicollinearity in logistic regression, but David Garson's Logistic Binomial Regression states that there is no valid test for multicollinearity for binary-dependent logistic regression, even if the independent variables are ratio scale. Can anyone supply one or more references? My own experience is that OLS correlation matrices and VIF worked for me, as my logistic coefficients went haywire before removing entangled independent variables based on the OLS tests for multicollinearity. But I have to publish my results and methods, and would like a reputable way to cite the practice, if one or more exist.
19801
I'm trying to predict a binary outcome using 50 continuous explanatory variables (the range of most of the variables is $-\infty$ to $\infty$). My data set has almost 24,000 rows. When I run glm in R, I get:  Warning messages:   1: glm.fit: algorithm did not converge   2: glm.fit: fitted probabilities numerically 0 or 1 occurred    I've read the other responses that suggest perfect separation might be occurring, but I'm confident that isn't the case in my data (though quasi-complete separation could exist; how can I test to see if that's the case?). If I remove some variables, the "did not converge" error might go away. But that's not always what happens.  I tried using the same variables in a bayesglm function and got the same errors.  What steps would you take to figure out exactly what's going on here? How do you figure out which variables are causing the problems?
39643
Since we are using the logistic function to transform a linear combination of the input into a non-linear output, how can logistic regression be considered a linear classifier?  Linear regression is just like a neural network without the hidden layer, so why are neural networks considered non-linear classifiers and logistic regression is linear?
78069
I'm trying to understand the derivation of the equations for the logistic regression. I'm following the cs229 notes:    At some point in the derivation, in the notes we need to maximize the log-likelihood of the parameters:  ℓ(θ) = log L(θ) = y log h(x) + (1 - y) log(1-h(x))  In the notes, after the derivation we get:  $\frac{\partial}{\partial \theta_j}$ = (y $\frac 1 {h(x)}$ - (1 - y) $\frac 1 {1 - h(x)}$) $\frac{\partial}{\partial \theta_j}$h(x)  Shouldn't this be:  $\frac{\partial}{\partial \theta_j}$ = (y $\frac 1 {h(x)}$ + (1 - y) $\frac 1 {1 - h(x)}$) $\frac{\partial}{\partial \theta_j}$h(x)  I've been looking at this for a while and I can't see where this minus sign is coming from...  Thanks for your help!
38281
Why is it that logistic regression becomes unstable when classes are well-separated? What does well-separated classes mean? I would really appreciate if someone can explain with an example.
92430
I'm exploring the effects of removing the intercept in a logistic regression model.  Assume a model:  $$logit(Y = 1) = \beta_1 x + \beta_2z + 0$$  with $x$ and $z$ being categorical variables with 2 levels each and no intercept.  I understood that having no intercept with categorical predictors produce coefficients that compare the $P(Y = 1)$ in each level of the two predictor against a null case where $P(Y=1) = 0.5$ or $logit(Y=1) = 0$.  I noticed a phenomenon that can understand. Using glm() function in R if you change the order of the variable in the right hand part of the formula, the coefficients change too. But even more oddly, the coefficient of the first variable is always the same.  Here's an R demo:  y &lt;- as.factor(sample(rep(1:2), 30, T)) x &lt;- as.factor(sample(rep(1:2), 30, T)) z &lt;- as.factor(sample(rep(1:2), 30, T))  coef(glm(y ~ x + z - 1, binomial) #        x1         x2         z2  #-0.1764783  0.3260739 -0.1335192  coef(glm(y ~ z + x - 1, binomial)) #        z1         z2         x2  #-0.1764783 -0.3099976  0.5025523    As you can see the first predictors have the same coefficient while the other are different in the two models.  Here is what I expected and instead behave differently than what I though:   Since every level of the two predictors is compared to the same null case, I expected to have the same coefficients in the two models, independently from the order in which I use them. I expected to see the coefficients of every level of every predictor, instead the coefficient for the 1 level of the second predictor is not shown. I therefore assume that only the first variable is compared against the null case, while the second is compared against a reference level; but what is this level? Is it $P(Y = 1 | X = 1 \cap Z = 1)$? Reproducing one of the models WITH the intercept we get:  coef(glm(y ~ x + z - 1, binomial) #        x1         x2         z2  #-0.1764783  0.3260739 -0.1335192  coef(glm(y ~ x + z, binomial)) #(Intercept)         x2          z2  #-0.1764783   0.5025523  -0.1335192    As expected x1 become the intercept, and x2 is likely relative to x1. z1 is missing also in this case and z2 is the same as in the model without intercept.  Thus should I assume that the comparison against the null case $P(Y = 1) = 0.5$ is made only for the first variable in a formula, while the other are compared against the usual intercept? Is this behavior normal? What about the fact that the first coefficient has the same value whichever the order of the predictors in the formula? What if I want to compare all level of each predictor against the null case and have a coefficient for all levels? Or it's theoretically impossible for some reason I don't get?
116146
From this , the following statement is posed: 'Though not "wrong", you'd want a good reason for using an identity link to model a Bernoulli probability.'  I would like to know what good reasons would result in an identity link being used for logistic regression, if these exist. Do these reasons generalise to GLMs?
28918
I proceeded by finding $\operatorname E(\bar X).$ I considered $\bar X$ as a constant and simply got the term itself. This should suggest that $\bar X$ is not an unbiased estimator of $\theta$. That's obvious too cuz it's a constant(from what I assumed). However, I'm also aware that $\mu = \theta$ so that's probably not right.  I have to demonstrate first that $\bar X$ is an unbiased estimator of $\mu$.
110823
I am trying to predict what are the determinants for foreign direct investments in Lithuanian regions from 2011 to 2017.   My dependent variable is foreign direct investments per capita in Lithuanian regions. And I have a variety of explanatory variables such as unemployment rate, labor cost, infrastructure quality and so on.   However, the data is really not normally distributed. Data does not show any linearity.     That is why I cannot use the multiple linear regression.     I found out that for the data which has a Poisson distribution, two regressions are suitable: Poisson and Negative binomial regression.   According to my calculations the dispersion is higher that the mean, so I should use the negative binomial regression.   Please correct me if I am wrong.   Also, how do you account for changing variables in time? (From 2011 to 2017)
100600
I have a very basic question, but it's not really clear to me. When would one choose logistic regression over SVM? Maximum margin property seems more justifiable then whatever log. reg does. Is it because scores or interoperability?
97309
In linear regression, I've seen (granted, not many) situations where basic transformations to some of the predictors can significantly improve the fit and stability of the model, and often a scatterplot of the response variable vs each individual predictor can provide useful clues as to whether a transformation can help.   Is there a similar approach that would work for logistic regression -  i.e., what is an intuitive way to determine whether any predictor transformations can help?
43552
Fitting a logistic regression using  ends with   Error in mer_finalize(ans) : Downdated X'X is not positive definite.    A likely cause of this error is apparently rank deficiency. What is rank deficiency, and how should I address it?