Say I have a multilinear model $y = X\beta - \varepsilon_1$, let's call this A. I also have another multilinear model $y = X_i\gamma - \varepsilon_2$, let's call this B. Where $X_i$ is a subset of $X$, meaning that all the variables in B are also found in A. According to some readings, as we increase the number of predictor variables, the SSR will almost always decrease. Can we prove this mathematically by showing that that $SS_{res}(A) \le SS_{res}(B)$.
68816
In my econometric textbook(Introductory Econometrics) covering OLS, the author write, "SSR must fall when another explanatory variable is added."  Why is it?
94588
We know that due to multi-collinearity, the standard errors of beta estimates get inflated. But what is the mathematical basis to it?   I am looking for some mathematical relationship or something to explain this.   Like I understand if standard error of betas goes up, then t-statistics goes down and we might not be able to reject the null or the variables would appear non-significant.   But what is the mathematical relationship between multicolllinearity and inflation in variance of coefficients?
35887
In a simple regression model,   $$ y = \beta_0 + \beta_1 x + \varepsilon, $$  the OLS estimators $\hat{\beta}_0^{OLS}$ and $\hat{\beta}_1^{OLS}$ are correlated.   The formula for the correlation between the two estimators is (if I have derived it correctly):  $$ \operatorname{Corr}(\hat{\beta}_0^{OLS},\hat{\beta}_1^{OLS}) = \frac{-\sum_{i=1}^{n}x_i}{\sqrt{n} \sqrt{\sum_{i=1}^{n}x_i^2} }. $$  Questions:   What is the intuitive explanation for the presence of correlation?   Does the presence of correlation have any important implications?   The post was edited and the assertion that the correlation vanishes with sample size has been removed. (Thanks to @whuber and @ChristophHanck.)
58379
I just got some question about the hat matrix in linear models.  My first question is: Why in a balanced one-way layout $(n_1=...=n_c=n_0)$, all leverages $h_{ii}$ have the same value $\frac{1}{n_0}$? I know that $h_{ii}$ is the $(i,i)$ entry in the hat matrix $H=X(X^TX)^{-1}X^T$.  But I couldn't see any relationship between this expression and the result.   My second question is:  when discussing the leverage points in a general linear model, we know that  $$0\leq h_{ii}\leq1$$ $$\sum_{i=1}^n h_{ii}=p$$ where $p$ is the number of parameters to be estimated, and $$Var(\hat {Y_i})=\sigma^2h_{ii}=\frac{\sigma^2}{\frac{1}{h_{ii}}}$$ Then it says $\frac{1}{h_{ii}}$ is roughly the number of observations needed to estimate $\hat{Y_i}$. Why is that?   My third question is:   Continuing with my second question, it then says, if $h_{ii}$ is very close to 1, then variance of the $i$th residual very lose to zero (already know that $Var(E_i)=\sigma^2(1-h_{ii})$)  so $Y_i-\sum_{j=1}^p\hat{\beta_j}x_{ij}\simeq0$.  Isn't this expression for the $i$th residual? Why we can conclude this from 0 variance of residual? I mean, can't $E_i$ be constant so that the variance is also zero?   My last question is:  Continuing with the above, it then conclude that $\hat{Y_i}\simeq Y_i$ and hence almost one degree of freedom needs to be used to just fit this one observation.  Could anyone explain this result to me? Isn't $\hat{Y_i}=Y_i$ exactly what we want?  Since I think it means we estimate the $i$th observation perfectly.
77107
The problem comes from linear regression. Assume the regression function is linear, i.e.  $$ f(X) = \beta_0+\sum_{j=1}^pX_j\beta_j $$ .Given a set of training data $(x_1, y_1),\ldots,(x_N,y_N)$,we try to estimate the parameters $\beta$ by minimizing the residual sum of squares:  $$\text{RSS}(\beta) = (\sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^px_{ij}\beta_j)^2$$ Here each $x_i=(x_{i1},\ldots,x_{ip})^T$ is vector in $\mathbb{R}^p$,$y_i\in\mathbb{R}$,and $\beta=(\beta_0,\ldots,\beta_{p+1})$. Let $\mathbf{X}$ denote matrix $(x_1,\ldots,x_N)^T$, $\mathbf{y}=(y_1,\ldots,y_N)$ and assume it has full column rank, it's easy to get $$\hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$ Suppose $\mathbf{\hat{y}}=\mathbf{X}\hat{\beta}$, and $\mathbf{y}=\mathbf{X}\beta$ for some $\beta$, to pin down the sampling properties of $\hat{\beta}$, now assume that the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and that the $x_i$ are fixed(non random). It's easy to see that $$\text{Var}(\hat\beta) = (\mathbf{X}^T\mathbf{X})^{-1}\sigma^2$$ ,so we turn to estimating the $\sigma^2$, and now comes the question: Let $\hat\sigma$ defined as below: $$\hat\sigma^2=\frac{1}{K}\sum_{i=1}^N(y_i-\hat{y_i})^2$$ It is said that only when $K=N-p-1$ did $\hat\sigma$ be an unbiased estimator of $\sigma$. But how do $p$(the dimension of $x$) be introduced in?
104108
Consider multiple linear regression. This question might be deceptively simple, but I'm trying to intuitively understand why, say if I have predictors X1 and X2, then interactions between these predictors can be adequately captured by X1 * X2.  I know interaction terms are modelled as products, just because that's what I was taught in school, and that's what everyone says to do.  I'm guessing there is maybe some geometric argument.  But why is a product (of say two numeric features, and not the extra complexity of multiplying by one being a dummy variable while other is numeric etc) going to adequately capture interactions?  Why are not "interactions" best captured by another f(X1, X2) by default instead of specifically X1 * X2?  I can see the idea that X1 * X2 may capture situations where the signs of X1 and X2 are the same or not, but then why would not, say, by default interactions be modelled by say f(X1, X2) = sign(X1) * sign(X2) instead of f(X1, X2) = X1X2?  I realise I can add any other f(X1, X2) to a regression or any predictive model, but finding the exact shape of interactions by hand coding is time consuming. How do I know X1X2 is a good first guess?
73646
I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of bedrooms are correlated. When predicting the cost of a house using these two predictors, one of them can be dropped because they are both providing a lot of the same information. Intuitively, this makes sense, but I have a some more technical questions:   How does this effect manifest itself in p-values of the regression coefficients when including only one or including both predictors in the model? How does the variance of the regression coefficients get affected by including both predictors in the model or just having one? How do I know which predictor the model will choose to be less significant? How does including only one or including both predictors change the value/variance of my forecasted cost?
12568
Is recasting a multivariate linear regression model as a multiple linear regression entirely equivalent?  I'm not referring to simply running $t$ separate regressions.   I have read this in a few places (Bayesian Data Analysis -- Gelman et al., and Multivariate Old School -- Marden) that a multivariate linear model can easily be reparameterized as multiple regression.  However, neither source elaborates on this at all.  They essentially just mention it, then continue using the multivariate model.  Mathematically, I'll write the multivariate version first,  $$ \underset{n \times t}{\mathbf{Y}} = \underset{n \times k}{\mathbf{X}} \hspace{2mm}\underset{k \times t}{\mathbf{B}} + \underset{n \times t}{\mathbf{R}}, $$ where the bold variables are matrices with their sizes below them.  As usual, $\mathbf{Y}$ is data, $\mathbf{X}$ is the design matrix, $\mathbf{R}$ are normally distributed residuals, and $\mathbf{B}$ is what we are interested in making inferences with.   To reparameterize this as the familiar multiple linear regression, one simply rewrites the variables as:  $$ \underset{nt \times 1}{\mathbf{y}} = \underset{nt \times nk}{\mathbf{D}}  \hspace{2mm} \underset{nk \times 1}{\boldsymbol{\beta}} + \underset{nt \times 1}{\mathbf{r}}, $$  where the reparameterizations used are $\mathbf{y} = row(\mathbf{Y}) $, $\boldsymbol\beta = row(\mathbf{B})$, and $\mathbf{D} = \mathbf{X} \otimes \mathbf{I}_{n}$.  $row()$ means that the rows of the matrix are arranged end to end into a long vector, and $\otimes$ is the kronecker, or outer, product.  So, if this is so easy, why bother writing books on multivariate models, test statistics for them etc.?  It is most effective to just transform the variables first and use common univariate techniques.  I'm sure there is a good reason, I just am having a hard time thinking of one, at least in the case of a linear model.  Are there situations with the multivariate linear model and normally distributed random errors where this reparameterization does not apply, or limits the possibilities of the analysis you can undertake?    Sources I have seen this: Marden - Multivariate Statistics: Old School.  See sections 5.3 - 5.5.  The book is available free from:   Gelman et al. - Bayesian Data Analysis.  I have the second edition, and in this version there is a small paragraph in Ch. 19 'Multivariate Regression Models' titled: "The equivalent univariate regression model"  Basically, can you do everything with the equivalent linear univariate regression model that you could with the multivariate model?  If so, why develop methods for multivariate linear models at all?  What about with Bayesian approaches?
153
In the simple linear regression model, $\hat{\beta}$ is the sum of independent normally distributed random variables.  Is it false because in linear regression there is $\beta$ and not $\hat{\beta}$?
40734
Suppose I have a random sample $\lbrace X_i, Y_i\rbrace_{i=1}^n$. Assume this sample is such that the Gauss-Markov assumptions are satisfied such that I can construct an OLS estimator where  $$\hat{\beta}_1^{OLS} = \frac{\text{Cov}(X,Y)}{\text{Var(X)}}$$ $$\hat{\beta}_0^{OLS} = \bar{Y} - \bar{X} \hat{\beta}_1^{OLS}$$  Now suppose I take my data set and double it, meaning there is an exact copy for each of the $n$ $(X_i,Y_i)$ pairs.  My Question  How does this affect my ability to use OLS? Is it still consistent and identified?
26091
In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to know $\hat\beta_0$ to estimate $\hat\beta_1$  Suppose I have $y=\beta_1x_1+\beta_2x_2$, how do I derive $\hat\beta_1$ without estimating $\hat\beta_2$? or is this not possible?
104790
Consider the following problem $$ \begin{array}{crl} &amp;\underset{y}{\max} &amp; f(y)\\ s.t. &amp; y_{1}A_{1}+y_{2}A_{2}+S_{1}&amp;=C_{1},\\  &amp; y_{3}A_{3}+y_{2}A_{4}+y_{5}A_{5}+S_{2}&amp;=C_{2},\\  &amp; y_{3}A_{6}+y_{4}A_{7}+S_{3}&amp;=C_{3},\\  &amp; S_{1},S_{2},S_{3}\succeq0, \end{array} $$ where $y_{i}$ are scalar variables and the matrices $A_{i}$ are symmetric real. How can we rewrite this optimzation problem in the following form  $$ \begin{array}{crl} &amp;\underset{y}{\max} &amp; f(y)\\ s.t. &amp; \sum_{i=1}^{m}y_{i}A_{i}+S&amp;=C,\\   &amp;S&amp;\succeq0, \end{array} $$ where $x_{i}$ are scalar variables and the matrices $B_{i}$ are symmetric real. please? Thanks.
75231
Suppose we have data matrix $X$ which is $n\times p$ matrix ($n$ data points, and $p$ features including intercept), i.e., we already add the intercept term (append $1$ column) to $X$   For logistic regression (binary classification), the model parameters / regression coefficients $\beta$ is a length $p$ vector. For softmax regression (multi-class classification), the model parameters $W$ is $p \times m$ matrix, where $m$ is the number of classes.   Now, suppose we set $m=2$, then $W$ is a $p \times 2$ matrix. They are the same model, so the number of parameters CANNOT be doubled and we should be able to derive one from another.   So, my question is  What is the relationship between $\beta$ and $W$?
88927
I have a model $p(x)$. How can adding latent variables $z$ help me? What are the main benefits of modelling $p(x, z)=p(x|z) p(z) $ instead of $p(x) $ alone? What would be some examples where modelling the latter would make my model better?
53948
Doing a little reading over the break (The Probabilistic Method by Alon and Spencer); can't come up with the solution for this seemingly simple (and perhaps even a little surprising?) result:     (A-S 1.6.3) Prove that for every two independent identically distributed real random variables $X$ and $Y$,   $$Pr[|X-Y| \leq 2] \leq 3 Pr[|X-Y| \leq 1].$$
40268
I am currently working to build a model using a multiple linear regression. After fiddling around with my model, I am unsure how to best determine which variables to keep and which to remove.  My model started with 10 predictors for the DV. When using all 10 predictors, four were considered significant. If I remove only some of the obviously-incorrect predictors, some of my predictors that were not initially significant become significant. Which leads me to my question: How does one go about determining which predictors to include in their model? It seemed to me you should run the model once with all predictors, remove those that are not significant, and then rerun. But if removing only some of those predictors makes others significant, I am left wondering if I am taking the wrong approach to all this.  I believe that  thread is similar to my question, but I am unsure I am interpreting the discussion correctly. Perhaps this is more of an experimental design topic, but maybe someone has some experience they can share.
121063
I have asked in another question how $\text{Var}{(\hat{y}_h)} = \sigma^2 \left(\frac{1}{n} + \frac{(x_h-\bar{x})^2}{S_{xx}}\right)$.  Note that $\hat{y}_h$ = $b_0 + b_1X_h$ which is a regression line estimate at some given $X_h$.  This question concerns why the term $Cov(b_0,b_1)$ alone yields the RHS.  Substituting $b_0 = Y - b_1X$ we get that $Cov(Y,b_1) - XCov(b_1,b_1)$ = $Cov(\frac{\sum{Y_i}}{n},\sum k_iY_i) - XVar{(b_1)}$. Here X and Y without subscript are arithmetic means.  We can then rearrange to obtain $\sum \frac{k_i Var(Y_i)}{n} - \frac{X\sigma^2}{S_{xx}}$ which quickly yields the desired result.  My question is, why does this work?  This single term does not seem like it should alone yield the RHS.  Have I made an error in algebra?