I know linear regression gets the estimate of beta by minimizing the squared errors. I'm curious why can't I use inverse of X to get the estimate of beta. I know there is errors and etc. but if X inverse exists, can we just estimate the beta using X ^(-1)? X * beta = Y beta = X^(-1) * Y This does not sound right, but I'm not sure how to disprove it either.
122705
I understand the derivation process of the least square estimator of B  𝑏=(𝑋′𝑋)-1  𝑋′𝑦  However, I am not seeing what I am doing wrong here, because I feel this can be further simplified like    𝑏 = (𝑋′𝑋)-1 𝑋′𝑦 = (𝑋)-1 (𝑋′)-1 𝑋′𝑦 = (𝑋)-1 Y  where am I doing wrong, please point out. Thank You!  Ok! I see , it has to be sqaure matrix to clearly define the inverse, I am just confused with derivation of the following:  Not sure how it gets from first red line to the second redline.. Thank you! Any help be will appreacited!
41097
I am interested in getting an unbiased estimate of $R^2$ in a multiple linear regression.  On reflection, I can think of two different values that an unbiased estimate of $R^2$ might be trying to match.    Out of sample $R^2$: the r-square that would be obtained if the regression equation obtained from the sample (i.e., $\hat{\beta}$) were applied to an infinite amount of data external to the sample but from the same data generating process. Population $R^2$: The r-square that would be obtained if an infinite sample were obtained and the model fitted to that infinite sample (i.e., $\beta$) or alternatively just the R-square implied by the known data generating process.   I understand that  is designed to compensate for the overfitting observed in sample $R^2$. Nonetheless, it's not clear whether adjusted $R^2$ is actually an unbiased estimate of $R^2$, and if it is an unbiased estimate, which of the above two definitions of $R^2$ it is aiming to estimate.  Thus, my questions:   What is an unbiased estimate of what I call above out of sample $R^2$? What is an unbiased estimate of what I call above population $R^2$? Are there any references that provide simulation or other proof of the unbiasedness?
53893
Say we fit a simple linear regression without the intercept term. I know this is inadvisable for the most part, unless there are some situations where it is reasonable to assume that when $x=0$, $y$ is also 0. I have been reading that when fitting without the intercept, it may cause the residual term to have non-zero mean. It makes sense but I can't seem to prove to myself (out of personal curiosity) why this is true. Can anyone help me understand it through a proof or some equations?
74140
In the univariate regression settings, we try to model   $$y = X\beta +noise$$  where $y \in \mathbb{R}^n$ a vector of $n$ observations and $X \in \mathbb{R}^{n \times m}$ the design matrix with $m$ predictors. The solution is $\beta_0 = (X^TX)^{-1}Xy$.  In the multivariate regression settings, we try to model   $$Y = X\beta +noise$$  where $y \in \mathbb{R}^{n \times p}$ is a matrix of $n$ observations and $p$ different latent variables. The solution is $\beta_0 = (X^TX)^{-1}XY$.  My question is how is that different than performing $p$ different univariate linear regression? I read  that in the latter case we take into consideration correlation between the dependent variables, but I don't see it from the math.
27828
Not sure if normalize is the correct word to use here, but I will try my best to illustrate what I am trying to ask. The estimator used here is least squares.  Suppose you have $y=\beta_0+\beta_1x_1$, you can center it around the mean by $y=\beta_0'+\beta_1x_1'$ where $\beta_0'=\beta_0+\beta_1\bar x_1$ and $x_1'=x-\bar x$, so that $\beta_0'$ no longer has any influence on estimating $\beta_1$.  By this I mean $\hat\beta_1$ in $y=\beta_1x_1'$ is equivalent to $\hat\beta_1$ in $y=\beta_0+\beta_1x_1$. We have reduced equation for easier least square calculation.  How do you apply this method in general? Now I have the model $y=\beta_1e^{x_1t}+\beta_2e^{x_2t}$, I am trying to reduce it to $y=\beta_1x'$.
59873
In a set of  that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this $$ y = \theta_1 + \theta_2\exp \left( {\theta_3x} \right) + \varepsilon $$ and suggests to first perform a nonlinear regression to estimate $\theta_3$, and then use OLS to estimate the other two parameters. I am not very experienced in regression modeling, so please forgive me if I ask a very simple question, but I don't understand how you can do this consistently without also estimating the other 2 parameters? Apparently, there is some trick involved, which I don't understand...  Another question is what happens to the ability to do inferences based on parameter estimates to the complete model if you do the regression in two stages like that?  Many thanks!
46340
In OLS regression, what does a one unit change in X represent in terms of a beta-related change in Y when the outcome variable has been transformed to $1/\sqrt(Y)$?   (Transformation was done to ensure linearity in the model. My predictor variables are 0-1 dummies.)  I've consulted the following useful posts, but I still don't know what is the answer:
41860
In simple linear regression, we have $y = \beta_0 + \beta_1 x + u$, where $u \sim iid\;\mathcal N(0,\sigma^2)$. I derived the estimator:  $$ \hat{\beta_1} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2}\ , $$ where $\bar{x}$ and $\bar{y}$ are the sample means of $x$ and $y$.   Now I want to find the variance of $\hat\beta_1$. I derived something like the following:  $$ \text{Var}(\hat{\beta_1}) = \frac{\sigma^2(1 - \frac{1}{n})}{\sum_i (x_i - \bar{x})^2}\ . $$  The derivation is as follow:  \begin{align} &amp;\text{Var}(\hat{\beta_1})\\ &amp; = \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \right) \\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} \text{Var}\left( \sum_i (x_i - \bar{x})\left(\beta_0 + \beta_1x_i + u_i - \frac{1}{n}\sum_j(\beta_0 + \beta_1x_j + u_j) \right)\right)\\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}  \text{Var}\left( \beta_1 \sum_i (x_i - \bar{x})^2 +  \sum_i(x_i - \bar{x}) \left(u_i - \sum_j \frac{u_j}{n}\right) \right)\\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\text{Var}\left( \sum_i(x_i - \bar{x})\left(u_i - \sum_j \frac{u_j}{n}\right)\right)\\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\;\times \\ &amp;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;E\left[\left( \sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n}) - \underbrace{E\left[\sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n})\right] }_{=0}\right)^2\right]\\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}  E\left[\left( \sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n})\right)^2 \right] \\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} E\left[\sum_i(x_i - \bar{x})^2(u_i - \sum_j \frac{u_j}{n})^2 \right]\;\;\;\;\text{ , since } u_i \text{ 's are iid} \\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\sum_i(x_i - \bar{x})^2E\left(u_i - \sum_j \frac{u_j}{n}\right)^2\\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\sum_i(x_i - \bar{x})^2 \left(E(u_i^2) - 2 \times E \left(u_i \times (\sum_j \frac{u_j}{n})\right) + E\left(\sum_j \frac{u_j}{n}\right)^2\right)\\ &amp; = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\sum_i(x_i - \bar{x})^2  \left(\sigma^2 - \frac{2}{n}\sigma^2 + \frac{\sigma^2}{n}\right)\\ &amp; = \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2}\left(1 - \frac{1}{n}\right) \end{align}  Did I do something wrong here?  I know if I do everything in matrix notation, I would get ${\rm Var}(\hat{\beta_1}) = \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2}$. But I am trying to derive the answer without using the matrix notation just to make sure I understand the concepts.
120459
Does the following qualify as a univariate regression?  $$y=b_0+b_1x+b_2x^2+\epsilon$$  I fully comprehend the implications of adding regressors and need no background information - a "yes" or "no" will do :-)
120436
I'm doing a linear regression with a transformed dependent variable. The following transformation was done so that the assumption of normality of residuals would hold. The untransformed dependant variable was negatively skewed, and the following transform made it close to normal:   $$Y=\sqrt{50-Y_{orig}}$$  where $Y_{orig}$ is the dependent variable on the original scale.   I think it makes sense to use some transformation on the $\beta$ coefficients to work our way back to the original scale. Using the following regression equation,   $$Y=\sqrt{50-Y_{orig}}=\alpha+\beta \cdot X$$  and by fixing $X=0$, we have   $$\alpha=\sqrt{50-Y_{orig}}=\sqrt{50-\alpha_{orig}}$$  And finally,  $$\alpha_{orig}=50-\alpha^2$$  Using the same logic, I found  $$\beta_{orig}=\alpha\space(\alpha-2\beta)+\beta^2+\alpha_{orig}-50$$  Now things work very well for a model with 1 or 2 predictors; the back-transformed coefficients resemble the original ones, only now I can trust the standard errors. The problem comes when including an interaction term, such as  $$Y=\alpha+X_1\beta_{X_1}+X_2\beta_{X_2}+X_1X_2\beta_{X_1X_2}$$  Then the back-transformation for the $\beta$s are not so close to the ones from the original scale, and I'm not sure why that happens. I'm also unsure if the formula found for back-transforming a beta coefficient is usable as is for the 3rd $\beta$ (for the interaction term). Before going into crazy algebra, I thought I'd ask for advice...
76957
This question is inspired from the long discussion in comments here:    In the usual linear regression model, for simplicity here written with only one predictor: $$      Y_i = \beta_0 + \beta_1 x_i + \epsilon_i $$ where the $x_i$ are known constants and $\epsilon_i$ are zero-mean independent error terms. If we in addition assume normal distributions for the errors, then the usual least squares estimators and the maximum likelihood estimators of $\beta_0, \beta_1$ are identical.  So my easy question: do there exist any other distribution for the error terms such that the mle are identical with the ordinary least squaeres estimator?  The one implication is easy to show, the other one not so.
61382
Beta regression (i.e. GLM with beta distribution and usually the logit link function) is often recommended to deal with response aka dependent variable taking values between 0 and 1, such as fractions, ratios, or probabilities: .  However, it is always claimed that beta regression cannot be used as soon as the response variable equals 0 or 1 at least once. If it does, one needs to either use zero/one-inflated beta model, or make some transformation of the response, etc.: .  My question is: which property of beta distribution prevents beta regression from dealing with exact 0s and 1s, and why?  I am guessing it is that $0$ and $1$ are not in the support of beta distribution. But for all shape parameters $\alpha&gt;1$ and $\beta&gt;1$, both zero and one are in the support of beta distribution, it's only for smaller shape parameters that the distribution goes to infinity at one or both sides. And perhaps the sample data are such that $\alpha$ and $\beta$ providing best fit would both turn out to be above $1$.  Does it mean that in some cases one could in fact use beta regression even with zeros/ones?   Of course even when 0 and 1 are in the support of beta distribution, probability of observing exactly 0 or 1 is zero. But so is the probability to observe any other given countable set of values, so this cannot be an issue, can it? (Cf.  by @Glen_b).  $\hskip{8em}$  In the context of beta regression, beta distribution is parameterized differently, but with $\phi=\alpha+\beta&gt;2$ it should still be well-defined on $[0,1]$ for all $\mu$.
18406
The standard error of the intercept term ($\hat{\beta}_0$) in $y=\beta_1x+\beta_0+\varepsilon$ is given by $$SE(\hat{\beta}_0)^2 = \sigma^2\left[\frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right]$$ where $\bar{x}$ is the mean of the $x_i$'s.  From what I understand, the SE quantifies your uncertainty- for instance, in 95% of the samples, the interval $[\hat{\beta}_0-2SE,\hat{\beta}_0+2SE]$ will contain the true $\beta_0$. I fail to understand how the SE, a measure of uncertainty, increases with $\bar{x}$. If I simply shift my data, so that $\bar{x}=0$, my uncertainty goes down? That seems unreasonable.  An analogous interpretation is - in the uncentered version of my data, $\hat{\beta}_0$ corresponds to my prediction at $x=0$, while in the centered data, $\hat{\beta}_0$ corresponds to my prediction at $x=\bar{x}$. So does this then mean that my uncertainty about my prediction at $x=0$ is greater than my uncertainty about my prediction at $x=\bar{x}$? That seems unreasonable too, the error $\epsilon$ has the same variance for all values of $x$, so my uncertainty in my predicted values should be the same for all $x$.  There are gaps in my understanding I'm sure. Could somebody help me understand what's going on?
56871
Can we say   $$\text{Var}(\beta_1) = \text{Var}\left(\frac{\sum (x_i-\bar x)y_i}{\sum (x_i- \bar x)^2}\right) = \left(\frac{\sum (x_i-\bar x)}{\sum (x_i- \bar x)^2}\right)^2 \text{Var}(y_i) \;\;??$$  I am not sure if I can separate the $x$'s i from $\sum (x_i-\bar x)y_i$. This expression seems like a linear combination of $y_i$'s. Is this legit because every $y_i$ follows the same distribution?
100652
In simple terms, what are the assumptions of Linear Regression?  I just want to know that when I can apply a linear regression model to our dataset.
37282
This seems to be a basic issue, but I just realized that I actually don't know how to test equality of coefficients from two different regressions. Can anyone shed some light on this?   More formally, suppose I ran the following two regressions:  $$ y_1 = X_1\beta_1 + \epsilon_1 $$ and  $$ y_2 = X_2\beta_2 + \epsilon_2 $$ where $X_i$ refers to the design matrix of regression $i$, and $\beta_i$ to the vector of coefficients in regression $i$. Note that $X_1$ and $X_2$ are potentially very different, with different dimensions etc. I am interested in for instance whether or not $\hat\beta_{11} \neq \hat\beta_{21}$.   If these came from the same regression, this would be trivial. But since they come from different ones, I am not quite sure how to do it. Does anyone have an idea or can give me some pointers?   My problem in detail: My first intuition was to look at the confidence intervals, and if they overlap, then I would say they are essentially the same. This procedure does not come with the correct size of the test, though (i.e. each individual confidence interval has $\alpha=0.05$, say, but looking at them jointly will not have the same probability). My "second" intuition was to conduct a normal t-test. That is, take   $$ \frac{\beta_{11}-\beta_{21}}{sd(\beta_{11})} $$  where $\beta_{21}$ is taken as the value of my null hypothesis. This does not take into account the estimation uncertainty of $\beta_{21}$, though, and the answer may depend on the order of the regressions (which one I call 1 and 2).   My third idea was to do it as in a standard test for equality of two coefficients from the same regression, that is take  $$ \frac{\beta_{11}-\beta_{21}}{sd(\beta_{11}-\beta_{21})} $$   The complication arises due to the fact that both come from different regressions. Note that   $$ Var(\beta_{11}-\beta_{21}) = Var(\beta_{11}) + Var(\beta_{21}) -2 Cov(\beta_{11},\beta_{21}) $$ but since they are from different regressions, how would I get $Cov(\beta_{11},\beta_{21})$?  This led me to ask this question here. This must be a standard procedure / standard test, but I cound not find anything that was sufficiently similar to this problem. So, if anyone can point me to the correct procedure, I would be very grateful!
114162
Goal: I have a function $f(x,y)=z$ (two variables for illustration only) which I know almost nothing about--it has a compact domain which I can determine, it is non-negative, and bounded above.  My goal is to find the maximum value $f(x,y)$ takes on its domain.  I can query the function and I know the values at a few points: $f(x_1, y_1)=z_1, f(x_2, y_2)=z_2,..., f(x_n, y_n)=z_n$.  Optimization via Linear Regression:  I have used linear regression (and related methods like the lasso) to model functions like this in the past; this is the method that I am very comfortable with.  Typically, I might try to find a polynomial (of degree $k$) which relatively accurately models the data. The end result would be to say that:   $$f(x,y)\approx \beta_1 + \beta_2 x + \beta_3 y + \beta_4 x^2 + \beta_5 xy + \beta_6 y^2 + ... + \beta_{m}y^{k}.$$  Determining the coefficients is done by solving the .  The "best degree" for the polynomial might be determined using $k$-fold cross validation.  Once the "best" polynomial has been determined, one can choose any of a number of methods to optimize it (Newton's method, gradient descent, hill climbing, etc).  Then, if all goes well, the optimal value on the polynomial corresponds to an optimal value of $f(x,y)$.  Bayesian Optimization:  Recently, I was describing this technique (linear regression) to a colleague who asked: "What is the difference between that and Bayesian Optimization?"  I've been attempting to answer this question for the last couple days, and so far I don't have anything really conclusive.  After reading , trying to read  and getting stuck reading  I think I understand the following about BO:   Start with a prior (a guess as to what family of functions the black box belongs to... or at least some family of functions which you believe might be able to describe $f$ well).  It looks using a  as a prior is popular. Specify whatever is needed to build your prior (for GP this is a covariance function). As you go along, you maintain two evolving functions, a "mean" function $m(x,y)$ and a "confidence interval" function $c(x,y)$.  Basically, you can predict the value at any point by evaluating $m(x,y)$ and you are confident that $f(x,y)\in [m(x,y)-c(x,y), m(x,y)+c(x,y)]$. Bayesian optimization provides a systematic way of updating/exploiting the model by alternatively selecting points where the confidence interval is wide (you are very uncertain about the value) or at optimal points on the model $m(x,y)$.   My Question: Is it fair/correct to say that the major difference between Bayesian Optimization and simply optimizing the linear regression model is that Bayesian Optimization provides the systematic way of sampling with the goal of optimizing the model and reducing uncertainty whereas the linear regression model ignores the idea of uncertainty and doesn't explicitly provide a means of updating the model?  Said another way, is the linear regression model an example of what might be a "mean" function $m(x,y)$ in Bayesian Optimization?  If I stick with linear regression, specify how to compute the uncertainty, and provide a means of updating/improving the model (exploit/explore) am I just doing Bayesian Optimization (without calling it that)?  Background: My background is in pure mathematics (graph theory) but my new job has me working with a lot of data (so I'm self learning statistics).  With that in mind, my statistics vocabulary is somewhat limited (so I appreciate answers that either avoid technical terminology or briefly explain any technical terminology used)--for example, Monday was the first time I heard/read the word "prior" used in a stats sense and I'm still not 100% sure I understand that correctly.