In the assumptions of linear regression, it is mentioned that error terms should be normally distributed with mean 0 and standard deviation $\sigma$. Is it necessary for the error terms to be normally distributed or just the latter part is enough to meet the assumptions?
122945
Seemingly reputable sources claim that the dependent variable must be normally distributed:     Model assumptions: $Y$ is normally distributed, errors are normally   distributed, $e_i \sim N(0,\sigma^2)$, and independent, and $X$ is fixed, and   constant variance $\sigma^2$.              Secondly, the linear regression analysis requires all variables to be   multivariate normal.              This is appropriate when the response variable has a normal   distribution         Is there a good explanation for how or why this misconception has spread? Is its origin known?  Related
113453
I'm taking an intro to statistics class and currently we are covering linear regression. The course is good but unfortunately it doesn't always gives all the information that is needed. For example it gives formulas for the slope and y-intercept as  $b=(r)\frac{s_Y}{s_X}=\frac{n \sum{xy} - \sum{x} \sum{y}}{n \sum{x^2} - (\sum{x})^2}$ , where  $r$ is the correlation coefficient  $s_Y$ is the standard deviation of the $Y$ scores  $s_X$ is the standard deviation of the $X$ scores   and  $a = \bar{y} - b \bar{x} = \frac{\sum{y} - b \sum{x}}{n}$  However it doesn't explain why the formulas are what they are. In other words it doesn't explain how these formulas where derived. Can someone show the derivation, especially the one for the slope, $b$, i.e. why $b=(r) \frac{s_Y}{s_X}$?  Any help is appreciated.
37344
I watched several videos on linear regression, mainly from . As I have no background in statistics, I thought this was a good way to get an idea of the topic. However I'm currently writing a bachelor thesis on Structural Equation Modeling and I want to get a deeper understanding of Regression Analysis.  The Book should roughly include these topics:   linear least squares regression variance, covariance regression coefficient coefficient of determination residual analysis (esp. leverage effect) multiple regression   I especially enjoyed the videos on khan academy, of the proof of minimizing the squared error to the regression line or illustrations like this one from Wikipedia:    I don't like it when books only present a formula and some common rules etc. without any further explanation of these.  F.e. Residuals should be normally distributed.  However I'm also not comfortable with strictly mathematical books, so I'm searching for something in between. A plus would be example code in R, but definitely not a must have.
100272
I am a little bit confused on what the assumptions of linear regression are. So far I checked whether:  all of the explanatory variables correlated linearly with the response variable. (This was the case) there was any collinearity among the explanatory variables. (there was little collinearity). the Cook's distances of the datapoints of my model are below 1 (this is the case, all distances are below 0.4, so no influence points). the residuals are normally distributed. (this may not be the case)  But I then read the following:  violations of normality often arise either because (a) the distributions of the dependent and/or independent variables are themselves significantly non-normal, and/or (b) the linearity assumption is violated.  Question 1 This makes it sound as if the independent and depend variables need to be normally distributed, but as far as I know this is not the case. My dependent variable as well as one of my independent variables are not normally distributed. Should they be? Question 2 My QQnormal plot of the residuals look like this:   That slightly differs from a normal distribution and the shapiro.test also rejects the null hypothesis that the residuals are from a normal distribution: &gt; shapiro.test(residuals(lmresult)) W = 0.9171, p-value = 3.618e-06  The residuals vs fitted values look like:   What can I do if my residuals are not normally distributed? Does it mean the linear model is entirely useless?
72308
I've been reading over this , trying to make sense of how the mean and variance of a gaussian conditional was derived. I've come to accept that unless I allocate a dozen or so hours to refreshing my linear algebra knowledge, it's out of my reach for the time being. that being said, I'm looking for a conceptual explanation for that these equations represent: $$\mu_{1|2} = \mu_1 + \Sigma_{1,2} * \Sigma^{-1}_{2,2}(x_2 - \mu_2)$$ I read the first as &quot;Take $\mu1$ and augment it by some factor, which is the covariance scaled by the precision (measure of how closely $X_2$ is clustered about $\mu_2$, maybe?) and projected onto the distance of the specific $x_2$ from $mu_2$.&quot; $$\Sigma_{1|2} = \Sigma_{1,1} - \Sigma_{1,2} * \Sigma^{-1}_{2,2} * \Sigma_{1,2}$$ I read the second as, &quot;take the variance about $\mu_1$ and subtract some factor, which is covariance squared scaled by the precision about $x_2$.&quot; In either case, the precision $\Sigma^{-1}_{2,2}$ seems to be playing a really important role. A few questions:  Am I right to treat precision as a measure of how closely observations are clustered about the expectation? Why is the covariance squared in the latter equation? (Is there a geometric interpretation?) So far, I've been treating $\Sigma_{1,2} * \Sigma^{-1}_{2,2}$ as a ratio, (a/b), and so this ratio acts to scale the (second) $\Sigma_{1,2}$, essentially accounting for/damping the effect of the covariance; I don't know if this is valid. Anything else you'd like to add/clarify?
95393
i. It is a known fact that unbiased estimator of the population variance is given by: $S^{2}=\dfrac{1}{n-1}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}$  This has no prior distribution assumption, so it should hold for any distribution of $X$.  ii. In linear regression, unbiased estimator of the error ($\epsilon$) variance is given by: $S^{2}=\dfrac{1}{n-p-1}\sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}$  which is the sum of squares of the residuals divided by the sample size minus the number of features.  My question is, why we don't use the formula given by (i) treating the residuals as a sample from the distribution of $\epsilon$? What is the difference between the two?
53893
Say we fit a simple linear regression without the intercept term. I know this is inadvisable for the most part, unless there are some situations where it is reasonable to assume that when $x=0$, $y$ is also 0. I have been reading that when fitting without the intercept, it may cause the residual term to have non-zero mean. It makes sense but I can't seem to prove to myself (out of personal curiosity) why this is true. Can anyone help me understand it through a proof or some equations?
73634
In practice, using a standard T-test to check the significance of a linear regression coefficient is common practice. The mechanics of the calculation make sense to me.  Why is it that the T-distribution can be used to model the standard test statistic used in linear regression hypothesis testing? Standard test statistic I am referring to here:   $$ T_{0} = \frac{\widehat{\beta} - \beta_{0}}{SE(\widehat{\beta})} $$
118525
Performing a regression and need to find out if my residuals are normally distributed.
87530
In many different statistical methods there is an "assumption of normality".  What is "normality" and how do I know if there is normality?
54410
Can someone briefly explain for me, why each of the six assumptions is needed in order to compute the OLS estimator? I found only about multicollinearity&mdash;that if it exists we cannot invert (X'X) matrix and in turn estimate overall estimator. What about the others (e.g., linearity, zero mean errors, etc.)?
8876
I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is not normally distributed, because this would result in normally distributed $y$. But when you compute the QQ-Normal plot there is evidence, that the residuals are normally distributed. Thus anyone can assume that the error term is normally distributed, although $y$ is not. So what does it mean, when the error term seems to be normally distributed, but $y$ does not?
83450
I went through this thread on degrees of freedom: , and the great answers in it, but then I read  on Wikipedia on the article about regression:     Statistical assumptions      When the number of measurements, N, is larger than the number of   unknown parameters, k, and the measurement errors εi are normally   distributed then the excess of information contained in (N − k)   measurements is used to make statistical predictions about the unknown   parameters. This excess of information is referred to as the degrees   of freedom of the regression.   Given this definition, if $N$ increases, the degrees of freedom increase as well, but intuitively that would make the problem more constrained (we have more information per parameter). Why is N-k then called degrees of freedom, and it isn't the other way around e.g. (k-N)?
53767
The question states:      Consider a set of random variables $X_i$, where $i=1,...n$. Each $X_i$ is   normally distributed with mean $0$ and variance $1$, i.e. $X_i$ are $\mathcal N(0,1)$.   What is the mean and the variance of the random variable $Y$, where   $Y=X_1+...+X_n$.   How do I do this?
116741
Is the following formula right if I want to measure the standard error of the median in case of a small sample with non normal distribution (I'm using python)?    sigma=np.std(data)  n=len(data)  sigma_median=1.253*sigma/np.sqrt(n)
118650
The question is very simple: why, when we try to fit a model to our data, linear or non-linear, do we usually try to minimize the sum of the squares of errors to obtain our estimator for the model parameter? Why not choose some other objective function to minimize? I understand that, for technical reasons, the quadratic function is nicer than some other functions, e.g., sum of absolute deviation. But this is still not a very convincing answer. Other than this technical reason, why in particular are people in favor of this 'Euclidean type' of distance function? Is there a specific meaning or interpretation for that?  The logic behind my thinking is the following:  When you have a dataset, you first set up your model by making a set of functional or distributional assumptions (say, some moment condition but not the entire distribution). In your model, there are some parameters (assume it is a parametric model), then you need to find a way to consistently estimate these parameters and hopefully, your estimator will have low variance and some other nice properties. Whether you minimize the SSE or LAD or some other objective function, I think they are just different methods to get a consistent estimator. Following this logic, I thought people use least square must be 1) it produces consistent estimator of the model 2) something else that I don't know.   In econometrics, we know that in linear regression model, if you assume the error terms have 0 mean conditioning on the predictors and homoscedasticity and errors are uncorrelated with each other, then minimizing the sum of square error will give you a CONSISTENT estimator of your model parameters and by the Gauss-Markov theorem, this estimator is BLUE. So this would suggest that if you choose to minimize some other objective function that is not the SSE, then there is no guarantee that you will get a consistent estimator of your model parameter. Is my understanding correct? If it is correct, then minimizing SSE rather than some other objective function can be justified by consistency, which is acceptable, in fact, better than saying the quadratic function is nicer.   In pratice, I actually saw many cases where people directly minimize the sum of square errors without first clearly specifying the complete model, e.g., the distributional assumptions (moment assumptions) on the error term. Then this seems to me that the user of this method just wants to see how close the data fit the 'model' (I use quotation mark since the model assumptions are probably incomplete) in terms of the square distance function.  A related question (also related to this website) is: why, when we try to compare different models using cross-validation, do we again use the SSE as the judgment criterion? i.e., choose the model that has the least SSE? Why not another criterion?
97693
Suppose we have two vectors of random variables, both are normal, i.e., $X \sim N(\mu_X, \Sigma_X)$ and $Y \sim N(\mu_Y, \Sigma_Y)$. We are interested in the distribution of their linear combination $Z = A X + B Y + C$, where $A$ and $B$ are matrices, $C$ is a vector. If $X$ and $Y$ are independent, $Z \sim N(A \mu_X + B \mu_Y + C, A \Sigma_X A^T + B \Sigma_Y B^T)$. The question is in the dependent case, assuming that we know the correlation of any pair $(X_i, Y_i)$. Thank you.  Best wishes, Ivan