Can someone please help me understand the above formula for prediction interval in the multi-variate context. seems to me X'X should be (X- \bar_X)'(X-\bar_X) where \bar_X is the mean vector. Since there are plenty of references for the formula when the number of "X" is "1". Also, seems like x_0 should be (x_0 -bar_X) as well. Is \hat_V_f missing an additional term "s^2/n"?
123433
What is the algebraic notation to calculate the prediction interval for multiple regression?  It sounds silly, but I am having trouble finding a clear algebraic notation of this.
116512
Suppose we have $X_1, \cdots, X_n \overset{\textrm{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ with $n &gt; 50$, and let $X_{(1)}, \cdots, X_{(n)}$ be the associated order statistics. Are there any references pointing to formulas that specify (or estimate) the variance of the $k^{\textrm{th}}$ order statistic, i.e. $\textrm{Var}(X_{(k)})$, for $1 \leq k \leq n$? I am looking for formulas can be explicitly evaluated without resorting back to numerical integration.  I have seen this question: , which attracted a number of answers on approximate formula for the mean of the $k^{\textrm{th}}$ order statistic, e.g. Blom (1958) (and modifications based on work by Harter (1961) and Elfving (1947)). Searching on Google also yields a number of work on the variance / standard deviation for the $k^\textrm{th}$ order statistics for small $n$, all of them opting for a computation/tabulation route. They are: Godwin(1949), Teichroew (1956), and Parrish (1992). I have also attempted to derive something myself following the method sketched by @probabilityislogic in , along the lines of: $$\textrm{Var}(X_{(k)}) = \int_{-\infty}^{\infty} \left(x-\mathbb{E}(X)\right)^2 \frac{n!}{(k-1)!(n-k)!} f_X(x)[1-F_X(x)]^{n-k}[F_X(x)]^{k-1}\;\textrm{d}x,$$ with lower case $f$ denoting the PDF of a r.v. $X$, and upper case $F$ denoting the CDF. Noting $x = F^{-1}(F_X(x))$ and using the substitution $u = F_X(x)$ (with corresponding r.v. in upper case $U$), we have: $$\textrm{Var}(X_{(k)}) = \int_{0}^{1} \left(F_X^{-1}(u) - \mathbb{E}\left(F_X^{-1}(U)\right)\right)^2\,\mathcal{B}(u | k, n-k+1) \; \textrm{d}u,$$ where $\mathcal{B}$ denotes the PDF of a beta distribution (not the beta function). This can be rewritten as the variance based on the beta distribution: $$\textrm{Var}(X_{(k)}) = \textrm{Var}_{\mathcal{B}(u | k, n-k+1)}\left(F_X^{-1}(u)\right).$$ Approximating the RHS using (the variance bit of the) delta method, we get: $$\textrm{Var}_{\mathcal{B}(u | k, n-k+1)}\left(F_X^{-1}(u)\right) \approx \textrm{Var}_{\mathcal{B}(u | k, n-k+1)}(U) \cdot \left[(F_X^{-1})'\left(\mathbb{E}_{\mathcal{B}(u | k, n-k+1)}(U)\right)\right]^2,$$ where the prime denotes the derivative. The first term of the product is standard result. For the second part, given $(F^{-1})'(\cdot)=\frac{1}{f(F^{-1})(\cdot)}$ (see e.g. ), we then have: $$\textrm{Var}_{\mathcal{B}(u | k, n-k+1)}\left(F_X^{-1}(u)\right) \approx \frac{k(n-k+1)}{(n+1)^2(n+2)} \frac{1}{\left(f\left(F^{-1}\left(\frac{k}{n+1}\right)\right)\right)^2}.$$ Substituting in the standard normal distribution ($\phi$ for PDF, $\Phi$ for CDF) and scaling it to the desired variance $\sigma^2$ we arrive at: $$\textrm{Var}(X_{(k)}) \approx \frac{k(n-k+1)}{(n+1)^2(n+2)} \frac{1}{\left(\phi\left(\Phi^{-1}\left(\frac{k}{n+1}\right)\right)\right)^2},$$ which we can throw in a $\sigma^2$ term in front if it is not a standard normal. But surely someone must have derived that somewhere (again I am looking for some references), and there is a high chance that I have gone off track somewhere above.
111655
I have the following data located . I am attempting to calculate the 95% confidence interval on the mean purity when the hydrocarbon percentage is 1.0. In R, I enter the following.   &gt; predict(purity.lm, newdata=list(hydro=1.0), interval="confidence", level=.95)    fit      lwr      upr 1 89.66431 87.51017 91.81845   However, how can I derive this result myself? I attempted to use the following equation.   $$s_{new}=\sqrt{s^2\left(1+\frac{1}{N}+\frac{(x_{new}-\bar x)^2}{\sum(x_i-\bar x)^2}\right)}$$  And I enter the following in R.   &gt; SSE_line = sum((purity - (77.863 + 11.801*hydro))^2) &gt; MSE = SSE_line/18 &gt; t.quantiles &lt;- qt(c(.025, .975), 18) &gt; prediction = B0 + B1*1 &gt; SE_predict = sqrt(MSE)*sqrt(1+1/20+(mean(hydro)-1)^2/sum((hydro - mean(hydro))^2)) &gt; prediction + SE_predict*t.quantiles [1] 81.80716 97.52146   My results are different from R's predict function. What am I misunderstanding about prediction intervals?
43054
I saw an example on a blog where a k nearest neighbors algorithm was run to predict responses on a test data set.The MSE(mean squared error) of the predictions was larger than the variance of the response variable in the test set. Shouldn't the MSE always be less than the variance of the response variable as SSTO(total sum of squares) = RSS(model sum of squares) + SSE(error sum of squares).
84532
I'm taking an Intro to Bayes course and I'm having some difficulty understanding predictive distributions. I understand why they are useful and I'm familiar with the definition, but there are some things I don't quite understand.   1) How to get the right predictive distribution for a vector of new observations  Suppose that we have built a sampling model $p(y_i | \theta)$ for the data and a prior $p(\theta)$. Assume that the observations $y_i$ are conditionally independent given $\theta$.   We have observed some data $\mathcal{D} = \{y_1, y_2, \, ... \, , y_k\}$, and we update our prior $p(\theta)$ to the posterior $p(\theta | \mathcal{D})$.   If we wanted to predict a vector of new observations $\mathcal{N} = \{\tilde{y}_1, \tilde{y}_2, \, ... \, , \tilde{y}_n\}$, I think we should try to get the posterior predictive using this formula $$ p(\mathcal{N} | \mathcal{D}) = \int p(\theta | \mathcal{D}) p ( \mathcal{N} | \theta) \, \mathrm{d} \theta = \int p(\theta | \mathcal{D}) \prod_{i=1}^n p(\tilde{y}_i | \theta) \, \mathrm{d} \theta, $$ which is not equal to $$ \prod_{i=1}^n  \int p(\theta | \mathcal{D}) p(\tilde{y}_i | \theta) \, \mathrm{d} \theta, $$ so the predicted observations are not independent, right?  Say that $\theta | \mathcal{D} \sim$ Beta($a,b$) and $p(y_i | \theta) \sim$ Binomial($n, \theta$) for a fixed $n$. In this case, if I wanted to simulate 6 new $\tilde{y}$, if I understand this correctly, it would be wrong to simulate 6 draws independently from the Beta-Binomial distribution that corresponds to the posterior predictive for a single observation. Is this correct? I don't know how to interpret that the observations are not independent marginally, and I'm not sure I understand this correctly.  Simulating from posterior predictives  Many times when we simulate data from the posterior predictive we follow this scheme:  For $b$ from 1 to $B$:  1) Sample $\theta^{(b)}$ from $p(\theta | \mathcal{D})$.  2) Then simulate new data $\mathcal{N}^{(b)}$ from $p(\mathcal{N} | \theta^{(b)})$.   I don't quite know how to prove this scheme works, although it looks intuitive. Also, does this have a name? I tried to look up a justification and I tried different names, but I had no luck.  Thanks!
83455
Assume we have $n$ distinct points $x_1,x_2,...,x_n\in \Bbb{R}$, and observations which are independent random variables $y_1,y_2,...,y_n$ depending on formula: $$ y_i=\theta_0+\theta_1x_i^2+\theta_3e^{x_i}+\varepsilon_i $$ where $\varepsilon_i$ are distributed as $N(0,\sigma^2)$. Derive the maximum likelihood estimator for $\theta_{0},\theta_{1},\theta_{2}$ .  I know how to find the MLE for a known distribution like normal or Poisson, but for this question, I don't how I can find the likelihood function, should I consider it as normal distribution? And I don't know what the mean and variance would be since I can't figure out the distribution and likelihood function. Also, should I consider $\varepsilon_i$ as a useful variable, or just treat it as an error in estimation? Thanks for any help.
28918
I proceeded by finding $\operatorname E(\bar X).$ I considered $\bar X$ as a constant and simply got the term itself. This should suggest that $\bar X$ is not an unbiased estimator of $\theta$. That's obvious too cuz it's a constant(from what I assumed). However, I'm also aware that $\mu = \theta$ so that's probably not right.  I have to demonstrate first that $\bar X$ is an unbiased estimator of $\mu$.
32266
What I am thinking was I need two formulas, AX.p(X) => something AX.p(X) => ~ something   I guess something maybe is the p(x) and the other is ~p(x) since we was given EX.~p(x)..But actually it can't work for me. And I guess maybe is EX.~p(x) and ~EX.~p(x)..But if I am doing that then the it will go to deadend..so I give up..;(  Wish someone can help me.   For note about EE: 1: EX.p(X) 2: AX.(p(X) => something) 3: something EE1,2 But something cann't have the free variable from p..in this sample, then soemthing cannot including X..so if something is p(X) it cannot use EE, but if p(Y) then can use EE.
100278
How do we go about calculating a posterior with a prior N~(a, b) after observing n data points? I assume that we have to calculate the sample mean and variance of the data points and do some sort of calculation that combines the posterior with the prior, but I'm not quite sure what the combination formula looks like.
97543
Suppose I have the following model: $$Y = \mu + \epsilon = X\beta + \epsilon,$$ where $Y$ is $n \times 1$, $X$ is $n \times p$, $\beta$ is $p \times 1$, and $\epsilon$ is $n \times 1$. I assume that $\epsilon$ are independent with mean 0 and variance $\sigma^2I$. In OLS, the fitted values are $\hat{Y} = HY$, where $H = X(X^TX)^{-1}X^T$ is the $N \times N$ hat matrix. I want to find the MSE of $\hat{Y}$. By the bias-variance decomposition, I know that \begin{align*} MSE(\hat{Y}) &amp;= bias^2(\hat{Y}) + var(\hat{Y})\\ &amp;= (E[HY] - \mu)^T(E[HY] - \mu) + var(HY)\\ &amp;= (H\mu - \mu)^T(H\mu - \mu) + \sigma^2H\\ &amp;= 0 + \sigma^2H \end{align*} I'm confused by the dimension in the last step. The $bias^2$ term is a scalar. However, $var(\hat{Y})$ is an $N \times N$ matrix. How can one add a scalar to an $N \times N$ matrix where $N \neq 1$?
35890
I am trying to generate two continuous predictors and one continuous outcome from mvrnorm. Now, I am trying to set up both standardized slopes at 0.3. This is my R code. n.tr=10 mu = c(3, 3, 3) Sigma = cbind(c(0.91,0,0.3),               c(0 0.91,0.3),               c(0.3,0.3,0.91) dat =  as.data.frame(mvrnorm(n=n.tr, mu=mu, Sigma=Sigma,                       empirical=TRUE)) colnames(dat) = c('z', 'w','y')  I want both of the standardized slopes to be 0.3, and 0 correlation between z and w. I also want the variance of y to be 1, so the residual variance of y is 0.91. If i want the diagonal of the matrix to be residual variance, how can I set up the matrix?
96308
I have a distribution $X$. By playing around with random samples from $X$, I've determined that $Var(X^i) &gt; Var(X)$ where $i &gt; 1$. However, I can't seem to find a formula for the expected variance of $X^i$, or why it should be greater than $Var(X)$.  Moving away from normal distributions, should it generalize that the scale parameter of any $X^i$ will be greater than $X$?
111978
I need a little help to move in the right direction.  It's a long time since I studied any stats and the jargon seems to have changed.  Imagine that I have a set of car-related data such as   Journey time from town A to town B Distance from town A to town B Engine size Driver's shoe size Make and model of car Day of week   I want to predict journey time.    I imagine there's a strong correlation between time and distance and probably a weaker one to engine size (and none to shoe size).  Presumably multiple regression analysis / ANOVA is the tool to use.  But how do I include day of week, since just coding it as Sunday=1, Monday=2 etc feels very wrong?  Having used Excel's regression tool, for example, how do I interpret the results?  Presumably if R is close to 1 this is good (although if there are many data items it seems as though it can be small yet still be significant).  But some sources refer to r-squared which seems to be the SD, so a value close to zero is good.  It also shows the t Stat, P-value, F and Significance F, whatever they may be.  Can anyone recommend a good reference source?
75191
In the book An introduction to statistical learning with Applications in R, when testing the relationship between the Response and the Predictors using null hypothesis, it gives a formula: $$F = \frac{(TSS - RSS)/p}{RSS/(n-p-1)}$$ where $$RSS = \sum_{i=1}^{n} (y_i-\hat{y_i})^2$$ $$\hat{y}=\hat{\beta_0} + \hat{\beta_1}x_1 + \dotsb + \hat{\beta_p}x_p$$ $$TSS = \sum_{i=1}^{n} (y_i-\bar{y})^2$$  I googled the F test and learned that the degree of freedom in F statistic means number of values in the final calculation of a statistic that are free to vary.(, )   But, here I cannot get why in this formula the degree of freedom are p and (n-p-1)?
60381
Notation: $\textbf{w}$ is an M-dimensional vector of parameters (including the bias parameter), $\textbf{x}_n$ is an M-dimensional vector of the features of each training example, $\textbf{t}$ is an N-dimensional vector of target values - where $t_n$ is the output of the nth training example and finally, $\Phi$ is an $M \times N$ design matrix.  During studying linear regression, I naturally came across the derivation of the normal equation. In his book, Christopher M. Bishop begins by the log of the likelihood function given by:  $$ \frac{N}{2} \ln \beta - \frac{N}{2} \ln(2\pi) - \frac{\beta}{2}\sum_{n=1}^N\{t_n - \textbf{w}^T \phi(\textbf{x}_n)\}^2 \tag{1} $$  With the goal of calculating the maximum likelihood of the parameters, he then calculates the derivative with respect to $w$:  $$ \bigtriangledown p(\textbf{t}| \textbf{w},\beta) = \beta \sum_{n=1}^N\{t_n - \textbf{w}^T \phi(\textbf{x}_n)\}\phi(\textbf{x}_n)^T \tag 2 $$  After which he sets the derivative to zero, in order to maximise the likelihood function  $$ 0 = \sum_{n=1}^N t_n \phi(\textbf{x}_n)^T - \textbf{w}^T \bigg(\sum_{n=1}^N \phi(\textbf{x}_n)\phi(\textbf{x}_n)^T \bigg) \tag 3 $$  So far so good. But then, the result for the maximum likelihood of the parameters $w$ is written as:  $$ \textbf{w}_{ML} = (\Phi^T\Phi)^{-1} \Phi^T \textbf{t} \tag 4 $$  My own derivation from equation $(3)$ to equation $(4)$ goes along the following lines:  \begin{align*} \Phi^T \Phi \textbf{w} &amp;= \Phi^T \textbf{t} \tag 5 \\  (\Phi^T \Phi)^{-1} \Phi^T \Phi \textbf{w} &amp;= (\Phi^T \Phi)^{-1} \Phi^T \textbf{t} \tag 6 \\ \textbf{w} &amp;= (\Phi^T \Phi)^{-1} \Phi^T \textbf{t} \tag 7 \end{align*}  My question is this: How does one go from equation $(3)$ to equation $(4)$. How does the supposed $ \textbf{w}^T \phi(\textbf{x}_n) \phi(\textbf{x}_n)$ become $\Phi^T \Phi \textbf{w}$ ? I do of course understand that given the dimensions of the matrix $\Phi$ and the vector $\textbf{w}$, that this is the only way to multiply the two entities. But what is the steps that one would follow in a formal derivation in order to reach such a result?
7965
This should be a fairly straight forward but I still couldn't quite get it. Let $X$ be $n$ by $1$ random vector, $Y$ be another $n$ by $1$ and that $X$ and $Y$ are independent. What is $Var(X^tY)=:V_{XY}$?  I tried generalizing this formula from the univariate case $Var(xy)=E(x)^2Var(y)+E(y)^2Var(x)+Var(x)Var(y)$  except I hit a bit of a roadblock for the 3rd term above. That is, I get $$ V_{XY}=X^tV_YX+Y^tV_XY\\+E[(X^t-\mu_X^t)(Y-\mu_Y)(Y^t-\mu_Y^t)(X-\mu_X)] $$ So... what is it?
84637
If I run a randomForest model, I can then make predictions based on the model.  Is there a way to get a prediction interval of each of the predictions such that I know how "sure" the model is of its answer.  If this is possible is it simply based on the variability of the dependent variable for the whole model or will it have wider and narrower intervals depending on the particular decision tree that was followed for a particular prediction?