My starting point is a covariance matrix (Not raw data) and I want to calculate the regression weights.  I may have as many as 6-8 independent variables and 1 dependent variable so looking for a matrix solution.  Been 20 years since I have done matrix algebra so any instructions for how to do this in Excel are appreciated.
18912
For simple linear regression, the regression coefficient is calculable directly from the variance-covariance matrix $C$, by  $$ C_{d, e}\over C_{e,e} $$ where $d$ is the dependent variable's index, and $e$ is the explanatory variable's index.  If one only has the covariance matrix, is it possible to calculate the coefficients for a model with multiple explanatory variables?   ETA: For two explanatory variables, it appears that $$\beta_1 = \frac{Cov(y,x_1)var(x_2) - Cov(y,x_2)Cov(x_1,x_2)}{var(x_1)var(x_2) - Cov(x_1,x_2)^2} $$ and analogously for $\beta_2$. I'm not immediately seeing how to extend this to three or more variables.
119923
It is common to use weights in applications like mixture modeling and to linearly combine basis functions. Weights $w_i$ must often obey $w_i ≥$ 0 and $\sum_{i} w_i=1$. I'd like to randomly choose a weight vector $\mathbf{w} = (w_1, w_2, …)$ from a uniform distribution of such vectors.  It may be tempting to use $w_i = \frac{\omega_i}{\sum_{j} \omega_j}$ where $\omega_i \sim$ U(0, 1), however as discussed in the comments below, the distribution of $\mathbf{w}$ is not uniform.  However, given the constraint $\sum_{i} w_i=1$, it seems that the underlying dimensionality of the problem is $n-1$, and that it should be possible to choose a $\mathbf{w}$ by choosing $n-1$ parameters according to some distribution and then computing the corresponding $\mathbf{w}$ from those parameters (because once $n-1$ of the weights are specified, the remaining weight is fully determined).  The problem appears to be similar to the  (but, rather than picking 3-vectors whose $ℓ_2$ norm is unity, I want to pick $n$-vectors whose $ℓ_1$ norm is unity).  Thanks!
4619
Consider the linear regression model:   $$Y_i=r(x_i)+\varepsilon_i\equiv\sum_{j = 1}^p x_{ij} \beta _j + \varepsilon _i,\quad i=1,\ldots,n.$$  where $x_1,\ldots,x_n\in \mathbb{R}^p$ are fixed, $E(\varepsilon_i)=0$,  $\operatorname{Var}(\varepsilon_i)=\sigma^2$. Denote $Y=(Y_1,\ldots,Y_n)^T$, $\beta=(\beta_1,\ldots,\beta_p)^T$, $X=(x_{ij})_{n\times p}$. As is known, $\hat \beta  = \arg\min\limits_{\beta  \in \mathbb{R}^p} (Y - X\beta)^T (Y - X\beta)=(X^TX)^{-1}X^TY$ if the matrix $X^TX$ is invertible, and so an estimator of $r(x)$ at $x=(x_1,\ldots,x_p)\in\mathbb{R}^p$ is given by $\hat r_n(x)=x^T\hat\beta$.  Let $$\hat{\sigma}^2 = \frac{1}{\sigma^2}\sum_{i = 1}^n (Y_i - {\hat r}_n (x_i))^2.$$ I am stucking the problem: $\hat\sigma^2$ has $\chi^2_{n-p}$ distribution. How to prove the statement?
37344
I watched several videos on linear regression, mainly from . As I have no background in statistics, I thought this was a good way to get an idea of the topic. However I'm currently writing a bachelor thesis on Structural Equation Modeling and I want to get a deeper understanding of Regression Analysis.  The Book should roughly include these topics:   linear least squares regression variance, covariance regression coefficient coefficient of determination residual analysis (esp. leverage effect) multiple regression   I especially enjoyed the videos on khan academy, of the proof of minimizing the squared error to the regression line or illustrations like this one from Wikipedia:    I don't like it when books only present a formula and some common rules etc. without any further explanation of these.  F.e. Residuals should be normally distributed.  However I'm also not comfortable with strictly mathematical books, so I'm searching for something in between. A plus would be example code in R, but definitely not a must have.
100442
According to linear regression formula, assuming the predictors are nonrandom, $${\rm var}[\hat{\beta}]=MSE *(X'X)^{-1}$$ where MSE is an unbiased estimator of the population variance.  Intuitively, when the number of predictors (p + 1) is greater than the number of data, the regression model is overfitting the data, variance of $\hat{\beta}$ should be big.  However, according to the formula, MSE = 0 and we have $${\rm var}[\hat{\beta}]=MSE *(X'X)^{-1} = 0$$  What goes wrong? There must be some implicit assumption with this formula that failed in this scenario.
99335
Say I have a collection of $n$ random variables $X_i$ that don't necessarily have any special properties like independence or identical distribution.  Is it true in general that $\text{Var}\left(\sum_{i=1}^n a_iX_i\right)=\sum_{i=1}^n\sum_{j=1}^n a_ia_j\text{Cov}\left(X_i,X_j\right)$, where the $a_i$ are constants?  I got to that equation by looking at the formula $\text{Var}\left(aX+bY\right)=a^2\text{Var}\left(X\right)+b^2\text{Var}\left(Y\right)+ab\text{Cov}\left(X,Y\right)$, viewing variance as a special case of covariance, and thinking about adding successively more variables.  If so, it looks like I could calculate the variance of a sum of random variables by adding up all the elements in their variance-covariance matrix--which would be interesting, since the combination of random variables itself is just a one-dimensional thing.  I think it clearly holds in the two-variable case, but I didn't find any discussion on the topic in the case of $n$ variables.
57852
In a regression fit of vectors varying with time $t$ $\qquad y \sim [x_t\ x_{t-1}\ x_{t-2}\ ...] \cdot [c_t\ c_{t-1}\ c_{t-2} \ ...] $ , how can one down-weight the older $x_t$ to model "older is less relevant" (than the regression gives) ? Say that rows in data matrix $X$ are observables (stocks, market segments ...) and columns are times. Correct me,   weights rows; how does one weight columns, times ?  Added: Down-weighting older data in time series looks complex, much more so than linear least squares with i.i.d. errors in rows (observations) -- just divide through by $\sigma_i$; or classification  with i.i.d. errors in columns (e.g. features) -- just centre each column. Can anyone say either   yes, I downweight time series; here are some examples on the web no: down-weighting time series is complex, not for novices.   (  discussed this question 10 years ago, mentioning Kalman filters and hierarchical Bayes, but with no examples.)
105865
Assume a p x n data matrix  (e.g. p independent samples of n variables). The n variables are correlated according to a correlation matrix $\rho$. Now if you consider the data matrix as some state t (t=time)  and based on this state I would like to calculate the next timestep (t+1). Therefore the correlation matrix would be allowed to have values &lt; 1 in the diagonal, which means that the diagonal elements would indicate the influence from each variable $x_{p,i}[t]$ on $x_{p,i}[t+1]$. So I would be interested to find a function f like this: $x_{p,i}[t+1] = f(x_{p,i}[t],\rho)$ so that the p x n data matrix  consisting of the $x_{p,i}[t+1]$ replicates the $\rho_{ij}$ (except the diagonal element). f should be something like this: $x_{i,j}[t+1] = \sum_{(k=1\ldots n)}\rho_{i,k} \cdot x_{kj}[t] + e_{i,j}.$ This means specifically, I not only want to calculate a data matrix, that replicates the correlation matrix as such, but also to let each observation be depending on its own state and linear dependent on the previous state of all the intercorrelated variables of its own state. For ease one can assume that the p variables are distributed normal and $e_{ij}$ are independent and distributed with mean$(e_{i,j}) = 0 $. Then for a numerical simulation it would be interesting  to find a formula to calculate the variance/standard deviation of each of the j variables (sd($e_{[.,j]} = ?).$ Thank you for any help!
113453
I'm taking an intro to statistics class and currently we are covering linear regression. The course is good but unfortunately it doesn't always gives all the information that is needed. For example it gives formulas for the slope and y-intercept as  $b=(r)\frac{s_Y}{s_X}=\frac{n \sum{xy} - \sum{x} \sum{y}}{n \sum{x^2} - (\sum{x})^2}$ , where  $r$ is the correlation coefficient  $s_Y$ is the standard deviation of the $Y$ scores  $s_X$ is the standard deviation of the $X$ scores   and  $a = \bar{y} - b \bar{x} = \frac{\sum{y} - b \sum{x}}{n}$  However it doesn't explain why the formulas are what they are. In other words it doesn't explain how these formulas where derived. Can someone show the derivation, especially the one for the slope, $b$, i.e. why $b=(r) \frac{s_Y}{s_X}$?  Any help is appreciated.
40177
Let $X,A,B,C,D$ be time-series variables and the covariance between any two pairs of these are known.  Suppose we want to find $\textrm{cov}(X,aA + bB + cC + dD)$, where $a,b,c,d$ are constants.  Is there any way of doing this without expanding out $E[(X-E[X])(aA+......)]$?
119620
Suppose, I have performed multiple regression analysis on the following data set where X1 and X2 are independent variables and Y is the dependent variable.    And achieved the following multiple regression analysis table from where it is clear that X2 is not correlated with Y as the p-value is greater than 0.05 which is 0.108 for X2 variable     So, if p value is greater than 0.05 for a variable, then why it is considered in regression equation  Y = 2.709 + 0.763(X1) + 0.463(X2) which has been shown in any online multiple regression equation calculator.  My question is Why it is not like the following Y = 2.709 + 0.763(X1) omitting the [0.463(X2)] part from the equation, as X2 is not correlated with Y  Please help to clarify the conceptual idea....
34198
I have an expression for a covariance matrix $C$ in terms of the indices $i$ and $j$.  In this way I can analytically calculate the elements of my covariance matrix, however when I try to invert $C$ matlab gives a warning about the matrix being close to singular.  The inversion therefore doesn't work, by which I mean the multiplying $C$ by the inverse given i do not get the identity.  I have tried calculating the pseudo inverse but this also does not work.  I have also tried adding a small constant along the diagonal but again the results do not work.  In general I'm working with matrices of dimension 1200 but I will give low dimensional example matrix that has the same properties, i.e. the matrices are symmetric about the diagonal and the anti-diagonal     19.9939  19.9954  19.9958  19.9951  19.9933  19.9905  19.9865    19.9954  19.9973  19.9981  19.9978  19.9965  19.9940  19.9905    19.9958  19.9981  19.9993  19.9995  19.9985  19.9965  19.9933    19.9951  19.9978  19.9995  20.0000  19.9995  19.9978  19.9951    19.9933  19.9965  19.9985  19.9995  19.9993  19.9981  19.9958    19.9905  19.9940  19.9965  19.9978  19.9981  19.9973  19.9954    19.9865  19.9905  19.9933  19.9951  19.9958  19.9954  19.9939   As mentioned in the title the matrix isn't positive, however the the negative eigenvalues are very small suggesting that the matrix is not positive only due to machine precision.  The negative eigenvalues are $-1.4048e-14$ and $-2.4571e-15$.  How can I go about inverting these matrices?
66997
Suppose I have a multiple linear regression of the form: $$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} +\epsilon$$ Then suppose I am interested in testing this set of hypotheses: $$H_0: \beta_1 = 0 \text{ OR } \beta_2 = 0$$ $$H_A: \beta_1 \neq 0 \text{ AND } \beta_2 \neq 0$$ I am interested in deriving a test statistic for this hypothesis.  I understand that I need to derive this under the null hypothesis.  However, it is not as straightforward as just plugging in $\beta_1 = 0 \text{ and } \beta_2 = 0$, since our null is that either/or is zero, not both. Any input would be much appreciated, thank you!
112001
I have a dataset with 10 variables ...is it feasible to transform each variable differently while doing multiple regression...  for example new_V1 = log(v1)  New_V2= V2^2  New_V3= 1/V3  Likewise differently for different variables and then applying multiple regression?
117574
I want to regress a variable $y$ onto $x,x^2,\ldots,x^5$. Should I do this using raw or orthogonal polynomials? I looked at the question on the site that deals with these, but I don't really understand what's the difference between using them. Why can't I just do a &quot;normal&quot; regression to get the coefficients $\beta_i$ of $y=\sum_{i=0}^5 \beta_i x^i$ (along with p-values and all the other nice stuff) and instead have to worry whether using raw or orthogonal polynomials? This choice seems to me to be outside the scope of what I want to do. In the stat book I'm currently reading (ISLR by Tibshirani et al) these things weren't mentioned. Actually, they were downplayed in a way. The reason is, AFAIK, that in the lm() function in R, using y ~ poly(x, 2) amounts to using orthogonal polynomials and using y ~ x + I(x^2) amounts to using raw ones. But on pp. 116 the authors say that we use the first option because the latter is &quot;cumbersome&quot; which leaves no indication that these commands actually do two completely different things (and have different outputs as a consequence). (third question) Why would the authors of ISLR confuse their readers like that?
58379
I just got some question about the hat matrix in linear models.  My first question is: Why in a balanced one-way layout $(n_1=...=n_c=n_0)$, all leverages $h_{ii}$ have the same value $\frac{1}{n_0}$? I know that $h_{ii}$ is the $(i,i)$ entry in the hat matrix $H=X(X^TX)^{-1}X^T$.  But I couldn't see any relationship between this expression and the result.   My second question is:  when discussing the leverage points in a general linear model, we know that  $$0\leq h_{ii}\leq1$$ $$\sum_{i=1}^n h_{ii}=p$$ where $p$ is the number of parameters to be estimated, and $$Var(\hat {Y_i})=\sigma^2h_{ii}=\frac{\sigma^2}{\frac{1}{h_{ii}}}$$ Then it says $\frac{1}{h_{ii}}$ is roughly the number of observations needed to estimate $\hat{Y_i}$. Why is that?   My third question is:   Continuing with my second question, it then says, if $h_{ii}$ is very close to 1, then variance of the $i$th residual very lose to zero (already know that $Var(E_i)=\sigma^2(1-h_{ii})$)  so $Y_i-\sum_{j=1}^p\hat{\beta_j}x_{ij}\simeq0$.  Isn't this expression for the $i$th residual? Why we can conclude this from 0 variance of residual? I mean, can't $E_i$ be constant so that the variance is also zero?   My last question is:  Continuing with the above, it then conclude that $\hat{Y_i}\simeq Y_i$ and hence almost one degree of freedom needs to be used to just fit this one observation.  Could anyone explain this result to me? Isn't $\hat{Y_i}=Y_i$ exactly what we want?  Since I think it means we estimate the $i$th observation perfectly.
42973
I know how to perform a linear regression on a set of points. That is, I know how to fit a polynomial of my choice, to a given data set, (in the LSE sense). However, what I do not know, is how to force my solution to go through some particular points of my choosing. I have seen this be done before, but I cannot remember what the procedure was called, let alone how it was done.   As a very simple and concrete example, let us say that I have 100 points scattered on the x-y plane, and I choose to fit a whatever-order polynomial through them. I know how to perform this linear regression very well. However, let us say that I want to 'force' my solution, to go through, say, three of my data points at x-coordinates $x=3$, $x=19$, and $x=89$, (and their corresponding y-coordinates of course).  What is this general procedure called, how is it done, and are there any particular pitfalls I need to be aware of?  Edit:  I would like to add, that I am looking for a concrete way to do this. I have written a program that actually does the linear regression in one of two ways, inverting the covariance matrix directly, or through gradient descent. What I am asking is, how, exactly, step by step, do I modify what I did, such that I force the polynomial solution to go through specific points?   Thanks!