In LOOCV, in general, the model has to be fit n times (n being the number of observations). The test error is then estimated as a mean of all the individual test errors.  Hastie and Tibshirani mention in section 5.1.2 of their  that with least squares linear or polynomial regression, the cost of LOOCV reduces to that of a single model fit, given by  $CV_{(n)} = \frac{1}{n}\sum_{i=1}^n (\frac{y_i - \hat{y_i}}{1 - h_i})^2$  where $\hat{y_i}$ is the $i$th fitted value from the original least squares fit, and $h_i$ is the leverage statistic.   Could you please explain how this result can be proved? Even after getting the estimate for the test error, how is the model chosen?
123587
From An Introduction to Statistical Learning by James et al., the leave-one-out cross-validation (LOOCV) estimate is defined by $$\text{CV}_{(n)} = \dfrac{1}{n}\sum\limits_{i=1}^{n}\text{MSE}_i$$ where $\text{MSE}_i = (y_i-\hat{y}_i)^2$.   Without proof, equation (5.2) states that for a  least-squares or polynomial regression (whether this applies to regression on just one variable is unknown to me), $$\text{CV}_{(n)} = \dfrac{1}{n}\sum\limits_{i=1}^{n}\left(\dfrac{y_i - \hat{y}_i}{1-h_i}\right)^2$$ where "$\hat{y}_i$ is the $i$th fitted value from the original least squares fit (no idea what this means, by the way, does it mean from using all of the points in the data set?) and $h_i$ is the leverage" which is defined by $$h_i = \dfrac{1}{n}+\dfrac{(x_i - \bar{x})^2}{\sum\limits_{j=1}^{n}(x_j - \bar{x})^2}\text{.}$$  How does one prove this?  My attempt: one could start by noticing that $$\hat{y}_i = \beta_0 + \sum\limits_{i=1}^{k}\beta_k X_k + \text{some polynomial terms of degree }\geq 2$$ but apart from this (and if I recall, that formula for $h_i$ is only true for simple linear regression...), I'm not sure how to proceed from here.
34980
The question has been asked (one time) on CV before, but the answer is really imprecise and does not really answer the question in my opinion.  So: What are the assumptions for estimating a linear regression model via quantile regression?  To my understanding (and as several CV users have mentioned), quantile regression does not assume any specific distribution of the error terms - does that mean that, in a time series model, autocorrelation and heteroscedasticity do not have to be accounted for?  What about the other Gauss-Markov assumptions? I would assume that the assumption of no perfect multicollinearity has to be met when applying quantile regression, but do the parameters have to be linear? The linearity assumption only has to hold for the specific quantile I would assume.  Anyways - I do not find any backup for any of my thoughts in the scientific literature and I would appreciate a comprehensive answer. Thank you!
59873
In a set of  that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this $$ y = \theta_1 + \theta_2\exp \left( {\theta_3x} \right) + \varepsilon $$ and suggests to first perform a nonlinear regression to estimate $\theta_3$, and then use OLS to estimate the other two parameters. I am not very experienced in regression modeling, so please forgive me if I ask a very simple question, but I don't understand how you can do this consistently without also estimating the other 2 parameters? Apparently, there is some trick involved, which I don't understand...  Another question is what happens to the ability to do inferences based on parameter estimates to the complete model if you do the regression in two stages like that?  Many thanks!
6777
I've read that it's bad to do this, but am looking for details as to why.  Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is continuous and $X$s are categorical with many levels. We don't have any opinion on how the levels should be grouped. We then calculate the average response for each level of the predictors, then collapse levels with similar averages.  How badly does this screw up parameter estimates and inferences on them, and why?
33015
I have been reading many articles on LASSO regression, and everyone claims that LASSO address multicollinearity showing contour plots of cost function touching the corner of the diamond(x1+x2). In the first place, I didn't understand how does this phenomenon translate to addressing multicollinearity when there are many groups of correlated variables in the data.  Verbatim from one of the research paper: "• If there are grouped variables (highly correlated between each other) LASSO tends to select one variable from each group ignoring the others".
116793
I'm a mathematician self-studying statistics and struggling especially with the language.   In the book I'm using, there is the following problem:  A random variable $X$ is given as $\text{Pareto}(\alpha,60)$-distributed with $\alpha&gt;0$. (Of course, you could take any distribution depending on one parameter for the sake of this question.) Then a sample of five values $14$, $21$, $6$, $32$, $2$ is given.   First part: "Using the method of maximum likelihood, find an estimate $\hat{\alpha}$ of $\alpha$ based on [the sample]." This was no problem. The answer is $\hat{\alpha}\approx 4.6931$.  But then: "Give an estimate for the standard error of $\hat{\alpha}$."  What is meant by this? Since $\hat{\alpha}$ is just a fixed real number, I don't see in what way it could have a standard error. Am I to determine the standard deviation of $\text{Pareto}(\hat{\alpha},60)$?  If you think the question is not clear, this information would help me as well.
39980
I'm looking at a regression model where a very large number of possible explanatory variables are being evaluated, and a small number are finally chosen via the lasso method of variable selection. The $\lambda$ tuning parameter in the lasso is chosen by looking at cross-validation forecast performance, which is pretty standard.   However, when I take the list of chosen variables and just run OLS on them, many turn out to be statistically insignificant. That may be perfectly fine if they are jointly significant and the forecast performance is superior to other models (in addition, there would be a question of what the t-test means when you have already screened the variables in a separate step, but I'm leaving that aside).   I'm curious though whether it makes sense to look at statistical significance of individual variables in a model chosen by lasso using CV forecast performance to select the tuning parameter. The problem is that lasso ends up selecting various dummy variables that are only true on small segments of the population and which are insignificant in OLS, and there is a natural question as to whether the model should be judgmentally simplified.
33650
Title says it all. I understand that the Least-Squares and Maximum-Likelihood will give the same result for regression coefficients if the model's errors are normally distributed. But, what happens if the errors are not normally distributed? Why are the two methods no longer equivalent?
37140
Does anyone have an intuitive explanation (no formulas, just words! :D) about the "$n-1$" instead of "$n$" in the unbiased variance estimator $$S_n^2 = \dfrac{\sum\limits_{i = 1}^n \left(X_i-\bar{X}\right)^2}{n-1}?$$
28105
Is it possible for more than one "linear regression" line to fit a given set of points? (i.e.... "least squares" is minimised equally in both cases)  I'm assuming a simple one-variable regression in which we have a single "dependent" variable for a given x-value. It may be that regression is forced "through the origin" for the purpose of the Q (if that matters), so, I know it wouldn't be a great fit.  Or is there something intrinsic about a regression line that means it is unique? I am interested in a 'proof' (could be just an intuitive proof) if it exists, that a regression line must be unique.
26091
In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to know $\hat\beta_0$ to estimate $\hat\beta_1$  Suppose I have $y=\beta_1x_1+\beta_2x_2$, how do I derive $\hat\beta_1$ without estimating $\hat\beta_2$? or is this not possible?
51797
Mboxcox in Stata suggests transforming my variables using a power of 0.1 for the independent variable, and 0.4 for the dependent variable.  I have run the model, and it fixes problems associated with the assumptions of OLS. But certainly, it complicates matters in terms of interpretations.  Please outline possible interpretations and solutions.  Dependent variable is in millions of dollars, and independent variables are number of years.  and a number of dummies which do not require transformation.
80621
I would like to estimate a multiple linear regression model $N$ observations (with $\beta$ of length $k$):  $$Y = X \beta + \epsilon$$  subject, however, to some linear constraints on the coefficients. I.e a constraint of the form  $$M \beta = 0$$  where (at least in my case) $M$ is of dimension $2 \times k$ but more generally can be expressed as  $$M \beta = c$$ where $M$ is of dimension $p \times k$ with $p &lt; N$ and $c$ of dimension $p \times 1$.  I have not been able to find much about this online and would appreciate references to read up on this..  Thanks
10777
$\newcommand{\Var}{\operatorname{Var}}$In this  is claimed that if the equation of errors in OLS is given by: $$u=y - X\beta$$ Then in the presence of heteroscedasticity the variance of $u$, will not be constant, $\sigma^2 \times I$, where $I$ is an identity matrix, but: $$\Var(u\mid X)=\sigma^2\Omega$$ In order to account for the heteroskedasticity, we can estimate the transform system, such that $P$ is a transformation matrix. $$Py=PX\beta-Pu$$ Where "the variance of a constant matrix $P$ times a random vector $u$" is: $$\Var(Pu\mid X)=P\Var(u\mid X)P'=P(\sigma^2\Omega)P'$$ Can somebody explain me the proof for that?
100442
According to linear regression formula, assuming the predictors are nonrandom, $${\rm var}[\hat{\beta}]=MSE *(X'X)^{-1}$$ where MSE is an unbiased estimator of the population variance.  Intuitively, when the number of predictors (p + 1) is greater than the number of data, the regression model is overfitting the data, variance of $\hat{\beta}$ should be big.  However, according to the formula, MSE = 0 and we have $${\rm var}[\hat{\beta}]=MSE *(X'X)^{-1} = 0$$  What goes wrong? There must be some implicit assumption with this formula that failed in this scenario.
95393
i. It is a known fact that unbiased estimator of the population variance is given by: $S^{2}=\dfrac{1}{n-1}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}$  This has no prior distribution assumption, so it should hold for any distribution of $X$.  ii. In linear regression, unbiased estimator of the error ($\epsilon$) variance is given by: $S^{2}=\dfrac{1}{n-p-1}\sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}$  which is the sum of squares of the residuals divided by the sample size minus the number of features.  My question is, why we don't use the formula given by (i) treating the residuals as a sample from the distribution of $\epsilon$? What is the difference between the two?
41895
I try to reproduce with optim the results from a simple linear regression fitted with glm or even nls R functions. The parameters estimates are the same but the residual variance estimate and the standard errors of the other parameters are not the same particularly when the sample size is low. I suppose that this is due differences in the way the residual standard error is calculated between Maximum Likelihood and Least square approaches (dividing by n or by n-k+1 see bellow in the example). I understand from my readings on the web that optimization is not a simple task but I was wondering if it would be possible to reproduce in a simple way the standard error estimates from glm while using optim.   Simulate a small dataset  set.seed(1) n = 4 # very small sample size ! b0 &lt;- 5 b1 &lt;- 2 sigma &lt;- 5 x &lt;- runif(n, 1, 100) y =  b0 + b1*x + rnorm(n, 0, sigma)    Estimate with optim  negLL &lt;- function(beta, y, x) {     b0 &lt;- beta[1]     b1 &lt;- beta[2]     sigma &lt;- beta[3]     yhat &lt;- b0 + b1*x     likelihood &lt;- dnorm(y, yhat, sigma)     return(-sum(log(likelihood))) }  res &lt;- optim(starting.values, negLL, y = y, x = x, hessian=TRUE) estimates &lt;- res$par     # Parameters estimates se &lt;- sqrt(diag(solve(res$hessian))) # Standard errors of the estimates cbind(estimates,se)       &gt; cbind(estimates,se)       estimates         se b0     9.016513 5.70999880 b1     1.931119 0.09731153 sigma  4.717216 1.66753138   Comparison with glm and nls  &gt; m &lt;- glm(y ~ x) &gt; summary(m)$coefficients             Estimate Std. Error   t value    Pr(&gt;|t|) (Intercept) 9.016113  8.0759837  1.116411 0.380380963 x           1.931130  0.1376334 14.030973 0.005041162 &gt; sqrt(summary(m)$dispersion) # residuals standard error [1] 6.671833 &gt;  &gt; summary(nls( y ~ b0 + b1*x, start=list(b0 = 5, b1= 2)))  Formula: y ~ b0 + b1 * x  Parameters:    Estimate Std. Error t value Pr(&gt;|t|)    b0   9.0161     8.0760   1.116  0.38038    b1   1.9311     0.1376  14.031  0.00504 ** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 6.672 on 2 degrees of freedom   I can reproduce the different residual standard error estimates like this :   &gt; # optim / Maximum Likelihood estimate &gt; sqrt(sum(resid(m)^2)/n) [1] 4.717698 &gt;  &gt; # Least squares estimate (glm and nls estimates) &gt; k &lt;- 3 # number of parameters &gt; sqrt(sum(resid(m)^2)/(n-k+1)) [1] 6.671833