I think this is a fairly technical, conceptual question so I'm going to do my best to explain what I'm thinking. For the regression $\widetilde Y = \hat \beta_0 + \hat \beta_1 X_1 + \hat \beta_2 X_2$, the equation for the coeficient $\hat \beta_2$ is $\text{cov}(Y,\widetilde X_2)/\text{var}(\widetilde X_2)$ where $\widetilde X_2$ are the $e$'s from the regression $X_2 = \hat \beta_0 + \hat \beta_1 X_1 + e$. Conceptually, these errors are the variance of $X_2$ that can't be explained by the other covariates, up to an affine transformation. Thus, only the &quot;unique variation&quot; in $X_2$ is what gives us additional power in predicting $Y$. The same can be said of $\hat \beta_1$ and $X_1$. What I'm trying to understand is what happened to the parts of the variance that were similar in $X_1$ and $X_2$? By similar I just mean that $X_2$ could explain in $X_1$ and what $X_1$ could explain in $X_2$. It seems like it needs to show up somewhere in the regression results but I can't find where it is... I think it'd end up in the $\hat \beta_0$ but I'm not sure if that's accurate or how to show it.
120760
So I asked  question and didn't get a response that cleared up my understanding. I came across some material today that provokes my question again in a different context. In some slides I'm going through, the creator included this diagram:  Which was referred to as &quot;Ballentine Graph&quot;. If we run the regression $\hat y = \hat \beta_0 + \hat \beta_1 X_1$ Then we'll recover the yellow and brown portion of this graph as $\hat \beta_1$. Similarly for $X_2$. If we run the regression $\hat y = \hat \beta_0 + \hat \beta_1 X_1 + \hat \beta_2 X_2$ then $\hat \beta_1$ will now only be the yellow area. Similar results for $X_2$. My question is, what happens to the brown area when $X_1$ and $X_2$ are both included in the regression? It'd be included in the coefficient recovered if $X_1$ and $X_2$ were included separately but it's not included in either if both regressors are included. Why does brown disappear from both regressors. Where in mathematical space did it go? To me, it seems like $X_1$ and $X_2$ would have to share it but since this isn't the case, did it go into $\hat \beta_0$? Why?
95393
i. It is a known fact that unbiased estimator of the population variance is given by: $S^{2}=\dfrac{1}{n-1}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}$  This has no prior distribution assumption, so it should hold for any distribution of $X$.  ii. In linear regression, unbiased estimator of the error ($\epsilon$) variance is given by: $S^{2}=\dfrac{1}{n-p-1}\sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}$  which is the sum of squares of the residuals divided by the sample size minus the number of features.  My question is, why we don't use the formula given by (i) treating the residuals as a sample from the distribution of $\epsilon$? What is the difference between the two?
85351
I am considering a linear regression model $Y_i = X_i^T \beta + \epsilon_i, i = 1,2,\dots,n$. where $X_i \in \mathbb{R}^p$. $\epsilon_i$'s are independent copies of random error $\epsilon \in \mathbb{R}^1$ with mean 0 and variance $\sigma^2$. We have the  hat matrix given by  $$ H = X^T (X^TX)^{-1} X$$ I am trying to show $h_{ij}^2 \leq h_{ii} h_{kk}$. I know that $$h_{ij} = X_i^T (X^T X)^{-1} X_j$$ and that $H^2 = H$ because it is idempotent. This gives me $\sum_{k=1}^n h_{ik}^2 = h_{ii}$ but I don't know how to prove $h_{ij}^2 \leq h_{ii} h_{jj}$.
76703
What is the distribution of the coefficient of determination, or R squared, $R^2$, in linear univariate multiple regression under the null hypothesis $H_0:\beta=0$?  How does it depend on the number of predictors $k$ and number of samples $n&gt;k$? Is there a closed-form expression for the mode of this distribution?  In particular, I have a feeling that for simple regression (with one predictor $x$) this distribution has mode at zero, but for multiple regression the mode is at a non-zero positive value. If this is indeed true, is there an intuitive explanation of this "phase transition"?    Update  As @Alecos showed below, the distribution indeed peaks at zero when $k=2$ and $k=3$ and not at zero when $k&gt;3$. I feel that there should be a geometrical view on this phase transition. Consider geometrical view of OLS: $\mathbf y$ is a vector in $\mathbb R^n$, $\mathbf X$ defines a $k$-dimensional subspace there. OLS amounts to projecting $\mathbf y$ onto this subspace, and $R^2$ is squared cosine of the angle between $\mathbf y$ and its projection $\hat{\mathbf y}$.  Now, from @Alecos's answer it follows that if all vectors are random, then the probability distribution of this angle will peak at $90^\circ$ for $k=2$ and $k=3$, but will have a mode at some other value $&lt;90^\circ$ for $k&gt;3$. Why?!    Update 2: I am accepting @Alecos'es answer, but still have a feeling that I am missing some important insight here. If anybody ever suggests any other (geometrical or not) view on this phenomenon that would make it "obvious", I will be happy to offer a bounty.
13237
It is often stated that the square of the sample correlation $r^2$ is equivalent to the $R^2$ coefficient of determination for simple linear regression. I have been unable to demonstrate this myself and would appreciate a full proof of this fact.
29759
Summary:  Am I correlating two independent variables?   Is that the problem?  Let's say I have data point for square footage of a house and the asking price.   Now, I can ask "Does square footage (x) determine price (y)?"  This is intuituve, and makes sense.  R-squared would end up saying "Square footage explains X% of the variation in price".  So far, so good.  But, what if I want to predict the square footage from the price?  That seems valid.  Now, I ask "Does price (x) determine square footage (y)?"  So far, it seems either can function as the independent or dependent variable.  However, the wording or r-squared seem off.  "Price explains X% of the variation in square footage".  Huh?  Square footage is not some sort of multi-factor variable.  It's more static.  Nothing "explains" the square footage, it just is.  Get what I'm saying?  Like if price only explains x% of square footage, what else would explain square footage?  Square footage is just square footage.  It's not like price, which can be determined by many things (square footage, renovations, size of yard, etc).  Another example can be age (x) and the mileage on a car (y).  With a regression equation, I can use one to predict the other.  Either order seems to work.  However does age "explain" the mileage, or does mileage "explain" the age?  In this case, both seem weird.  Both are just static independent variables.  Neither explains the other, if you ask me.  What am I missing here?  Thanks!
69134
Given is a linear regression problem, where we have one training point, which is 1-dimensional: $x \in R_{&gt;0}$ and the corresponding output, $y \in R$. We duplicate the feature, such that we have one training point with two (identical) features.   For this, we have to determine if we can apply the closed form solution $\beta = (X^TX)^-1*X^T*y$. Then we have to solve the linear regression problem by taking into account that $f(X) = ||y-X*\beta||^2$ is convex.   Application of the closed form solution: For this I want to determine if $X^TX$ has full rank. Given is $X = (1,x_{11},x_{12})$. Hence $X^T*X$ results in:  \begin{bmatrix}     1 &amp; x_{11} &amp; x_{12}  \\     x_{11} &amp; x_{11}^2 &amp; x_{11}x_{12}  \\     x_{12}  &amp; x_{11}x_{12} &amp; x_{12}^2   \end{bmatrix}   This matrix seems to have full rank (independent column vectors), so the closed form solution is applicable. Correct? Is there another way to determine the rank without explicitly calculating $X*X^T$?   For minimizing $f(\beta)$ I would derive $||y-X*\beta||^2$ and set it equal to zero.
89364
I've found for my econometrics exams that if I forget the scalar notation, I can often save myself by remembering the matrix notation and working backwards. However, the following confused me.  Given the simple estimation  $$\hat{y_i} = \hat{\beta_0} + \hat{\beta_1}x_{i1}$$  How do we get from  $$\boldsymbol{\hat{\beta}} = \boldsymbol{(X'X)}^{-1}\boldsymbol{X'y} $$  to  $$\hat{\beta}_1 = \frac{\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^n(x_i-\bar{x})^2} $$  I get stuck at  $$\hat{\beta}_1 = \frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$$
61941
I think this is a really simple question so please bear with me - I just had my first class in regression and I'm a little confused about nomenclature/labeling.  Does anyone recommend some good weblinks that explain beginning linear regression really well?  There's a question I've been looking at for a while and I'm not sure how to do it (although I'm sure the solution is simple):  Show that the sample estimate $\hat{\mu}(X) = \frac{1}{n} \sum X_i$ is a least square estimator of $\mu$ for a variable $X$ given $X_1, \ldots, X_n$.  My first thought was,  $\mathrm{SSE} = \sum (\mu - \hat{\mu})^2$  But I'm not sure if thats right. I'm confused about what the beta is (is it n?) and I don't know if there are enough parameters to expand it.  Thanks so much for your patience and if this doesn't make sense, I can clarify more. Thanks!
2768
I have scoured around, reading posts on Cross Validated () and also looking at references including Dobson and McCullagh and Nelder, e.g.  so I am aware that this topic is well trodden. Nevertheless, I am trying to articulate and formalize my understanding of GLM and while several posts have helped me with that, I am conscious of gaps and the possibility of an unsound foundation of my understanding.   In simple linear regression we have some set of observations $(x_i, y_i)$ pairs and treat $y_i$ as a realization of a random variable, $Y_i$, distributed as $Y_i \sim N(\mu_i, \sigma^2)$. The means ($\mu_i$) depend on the predictor but the variance is constant. We model $\mu_i = \beta_0 + (\beta_1 \times x_i)$ (or matrix equivalent ) which I believe is the same as saying $Y_i = \mu_i + \epsilon_i$ with $\epsilon_i \sim N(0,\sigma^2)$. I am not sure but I think a correct way to state $\mu$ is $\mu_i = E[Y_i|X=x]$. Can someone confirm that?  Anyway, we might transform the response to achieve linearity (maybe taking logs) and in that case we are modelling $log(Y_i) = \alpha_0 + (\alpha_1 \times x_i) + \epsilon_i$ where $\epsilon_i$ now is assumed to have a log normal distribution.   We generalize by decomposing the model into:   A structural component ($\beta_0 + (\beta_1 \times x_i)$) A link g(.) A response distribution (or random component) (member of the exponential family - Guassian, bin,gamma etc)   Let's say that the observations are assumed to come from a distribution in the exponential family and to keep things simple assume that it is the Gaussian distribution. Again, the expected value comes out as $E[Y_i|X=x] = \mu_i$, but does so from the first derivative of the b(theta) term in the exponential family form () (pg 29). In the case where we assume a binomial distribution, the expected value comes out at np but it still comes from the first derivative of the b(theta) term when the distribution is expressed in the exponential family form. Note, I don't fully appreciate or understand the derivation of the mean or variance at this stage, perhaps someone could provide a layman's explanation?  Instead of modelling the mean as was done for simple linear regression, we now model a transformation of the mean so instead of saying $\mu_i = \beta_0 + (\beta_1 \times x_i)$ we are saying $g(\mu_i) = \eta_i = \beta_0 + (\beta_1 \times x_i)$ where g is some link function (invertible and differentiable). I think this is a key (yet still somewhat confusing to me) distinction between SLR and GLM. In SLR we transform the response ($y_i$) and model that, in GLM we transform the expected value ($\mu_i$ in the case of the Gaussian example) and model that. Another way of saying it is that in the SLR case we model $E[g(Y_i|X=x)] = \beta_0 + (\beta_1 \times x_i)$ but in the GLM world we are modelling $\eta_i = g[E(Y_i|X=x)] = \beta_0 + (\beta_1 \times x_i)$.   My question is around verifying my understanding and statement of the essence of the foundations of GLM and the differences between GLM and the traditional linear model. Thanks.
63384
I am interested in the geometric meaning of the multiple correlation $R$ and coefficient of determination $R^2$ in the regression $y_i = \beta_1 + \beta_2 x_{2,i} + \dots + \beta_k x_{k,i} + \epsilon_i $, or in vector notation,  $$\mathbf{y} = \mathbf{X \beta} + \mathbf{\epsilon}$$  Here the design matrix $\mathbf{X}$ has $n$ rows and $k$ columns, of which the first is $\mathbf{x}_1 = \mathbf{1}_n$, a vector of 1s that corresponds to the intercept $\beta_1$.   The geometry is more interesting in the $n$-dimensional subject space rather than in the $k$-dimensional variable space. Define the hat matrix:  $$\mathbf{H} = \mathbf{X \left(X^\top X \right)}^{-1} \mathbf{X}^\top$$  This is an orthogonal projection onto the column space of $\mathbf{X}$, i.e. the   through the origin spanned by the $k$ vectors representing each variable $\mathbf{x}_i$, the first of which is $\mathbf{1}_n$. Then $\mathbf{H}$ projects the vector of observed responses $\mathbf{y}$ onto its "shadow" on the flat, the vector of fitted values $\mathbf{\hat{y}} = \mathbf{Hy}$, and if we look along the path of the projection we see the vector of residuals $\mathbf{e} = \mathbf{y} - \mathbf{\hat{y}}$ forms the third side of a triangle. This should furnish us with two routes to a geometric interpretation of $R^2$:   The square of the multiple correlation coefficient, $R$, which is defined as the correlation between $\mathbf{y}$ and $\mathbf{\hat{y}}$. This will appear geometrically as the cosine of an angle. In terms of lengths of vectors: for instance $SS_\text{residual} = \sum_{i=1}^{n}e_i^2 = \|\mathbf{e}\|^2$.   I would be delighted to see a brief account which explains:   The finer details for (1) and (2), Why (1) and (2) are equivalent, Briefly, how the geometric insight lets us visualise basic properties of $R^2$, for instance why it goes to 1 when the noise variance goes to 0. (After all, if we can't intuit from our visualisation then it's no more than a pretty picture.)   I appreciate this is more straightforward if the variables are centred first, which removes the intercept from the question. However, in most textbook accounts which introduce multiple regression, the design matrix $\mathbf{X}$ is as I laid out. Of course it's fine if an exposition delves into the space spanned by the centred variables, but for insight into the textbook linear algebra, it would be very helpful to relate this back to what's happening geometrically in the uncentred situation. A really insightful answer might explain what exactly is breaking down geometrically  - i.e. when the vector $\mathbf{1}_n$ is removed from the spanning set. I don't think this last point can be addressed by considering the centred variables alone.
29900
Specifically, I want to know if there is a difference between lm(y ~ x1 + x2) and glm(y ~ x1 + x2, family=gaussian). I think that this particular case of glm is equal to lm. Am I wrong?
14012
Is it possible to have a (multiple) regression equation with two or more dependent variables?  Sure, you could run two separate regression equations, one for each DV, but that doesn't seem like it would capture any relationship between the two DVs?
71529
The following is the residuals vs predicted scatter plot for a regression model with two IVs.   Initially, I thought it was evidence of heteroskedasticity. But, I reasoned that although there is a visible pattern in the plot, the variance across different levels of predicted values is same.  To clear my doubts, I saved the standardized residual and predicted values and ran a bi-variate correlation test. The correlation is zero, as expected. I am, nevertheless, intrigued by this observation. Any idea why I might have obtained this pattern?  PS:- My dependent variable is a sum of two likert-type items (Response scale: 1-5). So, it's theoretical range is 2-10 and it has no absolute zero value.
28105
Is it possible for more than one "linear regression" line to fit a given set of points? (i.e.... "least squares" is minimised equally in both cases)  I'm assuming a simple one-variable regression in which we have a single "dependent" variable for a given x-value. It may be that regression is forced "through the origin" for the purpose of the Q (if that matters), so, I know it wouldn't be a great fit.  Or is there something intrinsic about a regression line that means it is unique? I am interested in a 'proof' (could be just an intuitive proof) if it exists, that a regression line must be unique.
32415
Usually regression models are evaluated using $R^2$. I understand this metric can be misleading too at times but as far as I understand the first parameter we look at is $R^2$.   There is another parameter which is often used and it is $MAPE$.  Both are functions of errors between predicted and true value. I am just wondering if there are certain cases where one should be preferred above another?   $R^2$ can become negative if being used on test data (on which the model is not built). Is this a reason to not use $R^2$ in these cases?  Can anyone give a qualitative insight on where $MAPE$ should be used and where $R^2$ should be used.
18406
The standard error of the intercept term ($\hat{\beta}_0$) in $y=\beta_1x+\beta_0+\varepsilon$ is given by $$SE(\hat{\beta}_0)^2 = \sigma^2\left[\frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right]$$ where $\bar{x}$ is the mean of the $x_i$'s.  From what I understand, the SE quantifies your uncertainty- for instance, in 95% of the samples, the interval $[\hat{\beta}_0-2SE,\hat{\beta}_0+2SE]$ will contain the true $\beta_0$. I fail to understand how the SE, a measure of uncertainty, increases with $\bar{x}$. If I simply shift my data, so that $\bar{x}=0$, my uncertainty goes down? That seems unreasonable.  An analogous interpretation is - in the uncentered version of my data, $\hat{\beta}_0$ corresponds to my prediction at $x=0$, while in the centered data, $\hat{\beta}_0$ corresponds to my prediction at $x=\bar{x}$. So does this then mean that my uncertainty about my prediction at $x=0$ is greater than my uncertainty about my prediction at $x=\bar{x}$? That seems unreasonable too, the error $\epsilon$ has the same variance for all values of $x$, so my uncertainty in my predicted values should be the same for all $x$.  There are gaps in my understanding I'm sure. Could somebody help me understand what's going on?