Why do &quot;most&quot; (basically all) statistics text books use $\hat{\sigma}^2=\hat{p} (1-\hat{p})$ as an estimator for the variance of a Bernoulli process which we know is . Should the default not be the bias corrected sample version $s^2 = \hat{p} (1-\hat{p}) \cdot n/(n-1)$, which is the default for non-binary data.
121905
$Y_{1...n}\sim \operatorname{Bin}(1,p)$, iid,  and I need to find an unbiased estimator for $\theta=\operatorname{var}(y_i)$.  I did some calculations and I think that the answer is $p(1-p)-\frac{p(1-p)}{n}$   Is this correct? If not, how can I find an unbiased estimator?
119610
Suppose that $X_1, \ldots, X_n$ are iid Bernoulli trials with parameter $p$, and that $Y = \sum_{i=1}^nX_i$. Then, we have that $Y$ is a sufficient statistic by the following condensed proof:  $$ P(X_1 = x_1,  ... , X_n = x_n |Y = y) = \frac{p^y(1-p)^{n-y}}{\binom{n}{y}  p^y(1-p)^{n-y}} =\frac{1}{\binom{n}{y}}   \text{ if } \sum_{i=1}^{n}x_i = y $$  I am trying to see why this is the case. My reasoning is that there is some cancellation property on top and bottom that "kills" off the parameter $p$. In that whatever is on top through $p$ is perfectly matched by whatever is on the bottom minus scaling factors (in this case $\binom{n}{y}^{-1}$).   However, I am wondering what the deeper meaning behind this is, that the numerator of the joint distribution of the data and the sufficient statistic has its EXACT form on the bottom, except for constants which are only multiplicative (to facilitate cancellations). Am I onto something bigger and deeper here?
77107
The problem comes from linear regression. Assume the regression function is linear, i.e.  $$ f(X) = \beta_0+\sum_{j=1}^pX_j\beta_j $$ .Given a set of training data $(x_1, y_1),\ldots,(x_N,y_N)$,we try to estimate the parameters $\beta$ by minimizing the residual sum of squares:  $$\text{RSS}(\beta) = (\sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^px_{ij}\beta_j)^2$$ Here each $x_i=(x_{i1},\ldots,x_{ip})^T$ is vector in $\mathbb{R}^p$,$y_i\in\mathbb{R}$,and $\beta=(\beta_0,\ldots,\beta_{p+1})$. Let $\mathbf{X}$ denote matrix $(x_1,\ldots,x_N)^T$, $\mathbf{y}=(y_1,\ldots,y_N)$ and assume it has full column rank, it's easy to get $$\hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$ Suppose $\mathbf{\hat{y}}=\mathbf{X}\hat{\beta}$, and $\mathbf{y}=\mathbf{X}\beta$ for some $\beta$, to pin down the sampling properties of $\hat{\beta}$, now assume that the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and that the $x_i$ are fixed(non random). It's easy to see that $$\text{Var}(\hat\beta) = (\mathbf{X}^T\mathbf{X})^{-1}\sigma^2$$ ,so we turn to estimating the $\sigma^2$, and now comes the question: Let $\hat\sigma$ defined as below: $$\hat\sigma^2=\frac{1}{K}\sum_{i=1}^N(y_i-\hat{y_i})^2$$ It is said that only when $K=N-p-1$ did $\hat\sigma$ be an unbiased estimator of $\sigma$. But how do $p$(the dimension of $x$) be introduced in?
80817
The formula for computing variance has $(n-1)$ in the denominator:  $s^2 = \frac{\sum_{i=1}^N (x_i - \bar{x})^2}{n-1}$  I've always wondered why. However, reading and watching a few good videos about "why" it is, it seems, $(n-1)$ is a good unbiased estimator of the population variance. Whereas $n$ underestimates and $(n-2)$ overestimates the population variance.  What I'm curious to know, is that in the era of no computers how exactly was this choice made?  Is there an actual mathematical proof proving this or was this purely empirical and statisticians made A LOT of calculations by hand to come up with the "best explanation" at the time?   Just how did statisticians come up with this formula in the early 19th century with the aid of computers? Manual or there is more to it than meets the eye?
36151
It's a standard result that given $X_1,\cdots ,X_n $ random sample from $N(\mu,\sigma^2)$, the random variable $$\frac{(n-1)S^2}{\sigma^2}$$ has a chi-square distribution with $(n-1)$ degrees of freedom, where $$S^2=\frac{1}{n-1}\sum^{n}_{i=1}(X_i-\bar{X})^2.$$ I would like help in proving the above result. Thanks.
50064
So there is  about  why variance  is squared.   And the answer seems to be "because we get to do groovy maths when it is squared". Ok, that's cool, I can dig.  However, I'm sitting reading some financial maths stuff, and a lot of the equations on pricing and risk are based on variance. It doesn't seem to be the best basis for, you know, pricing exotic vehicles that are worth in the millions (or billions) that a formula for variance is used "because the maths is better".    To make a point  then, why not have the variance be from the cubed, abs cubed, or the 4th power (or even a negative power)?   eg (apologies, I don't know Latex)  Sum 1/N * |(x - mean)^3|   OR  Sum 1/N * (x - mean)^4   Would using variance-to-a-different power measurably alter pricings/valuations if the equations still used variance as usual (but the variance was calculated with the different power)?  Is there a reason why we stopped  at "power of 2", and are there any implications of using a variance concocted from a different (higher or lower) power?
86539
Let $X_1,...,X_n$ be Bernoulli distributed with unknown parameter $p$.   My objective is to calculate the information contained in the first observation of the sample.  I know that the pdf of $X$ is given by $$f(x\mid p)=p^x(1-p)^{1-x}$$, and my book defines the Fisher information about $p$ as   $$I_X(p)=E_p\left[\left(\frac{d}{dp}\log\left(p^x(1-p)^{1-x}\right)\right)^2\right]$$   After some calculations, I arrive at  $$I_X(p)=E_p\left[\frac{x^2}{p^2}\right]-2E_p\left[\frac{x(1-x)}{p(1-p)}\right]+E_p\left[\frac{(1-x)^2}{(1-p)^2}\right]$$  I know that the Fisher information about $p$ of a Bernoulli RV is $\frac{1}{p(1-p)}$, but I don't know how to get rid of the X-values, since I'm calculating an expectation with respect to $p$, not $X$. Any clues?
42859
According to the Wikipedia article on  the sample SD   $$s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \overline{x})^2}$$  is a biased estimator of the SD of the population. It states that $E(\sqrt{s^2}) \neq \sqrt{E(s^2)}$.   NB. Random variables are independent and each $x_{i} \sim N(\mu,\sigma^{2})$   My question is two-fold:   What is the proof of the biasedness? How does one compute the expectation of the sample standard deviation    My knowledge of maths/stats is only intermediate.
81753
From this  on Bayesian Least Mean Square estimation:  $$\mathbf E\left[(\Theta -\hat\theta)^2\right] = \rm{var}(\Theta -\hat\theta) + \left(\mathbf E[\Theta -\hat\theta]\right)^2 = \rm{var}(\Theta) + \left(\mathbf E[\Theta -\hat\theta]\right)^2$$  which is to say that $\rm{var}(\Theta -\hat\theta) = \rm{var}(\Theta)$.  I would like to ask how that comes about?  Thanks.  $\Theta$ is the random variable being estimated, and $\hat\theta$ is a constant estimate.
116512
Suppose we have $X_1, \cdots, X_n \overset{\textrm{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ with $n &gt; 50$, and let $X_{(1)}, \cdots, X_{(n)}$ be the associated order statistics. Are there any references pointing to formulas that specify (or estimate) the variance of the $k^{\textrm{th}}$ order statistic, i.e. $\textrm{Var}(X_{(k)})$, for $1 \leq k \leq n$? I am looking for formulas can be explicitly evaluated without resorting back to numerical integration.  I have seen this question: , which attracted a number of answers on approximate formula for the mean of the $k^{\textrm{th}}$ order statistic, e.g. Blom (1958) (and modifications based on work by Harter (1961) and Elfving (1947)). Searching on Google also yields a number of work on the variance / standard deviation for the $k^\textrm{th}$ order statistics for small $n$, all of them opting for a computation/tabulation route. They are: Godwin(1949), Teichroew (1956), and Parrish (1992). I have also attempted to derive something myself following the method sketched by @probabilityislogic in , along the lines of: $$\textrm{Var}(X_{(k)}) = \int_{-\infty}^{\infty} \left(x-\mathbb{E}(X)\right)^2 \frac{n!}{(k-1)!(n-k)!} f_X(x)[1-F_X(x)]^{n-k}[F_X(x)]^{k-1}\;\textrm{d}x,$$ with lower case $f$ denoting the PDF of a r.v. $X$, and upper case $F$ denoting the CDF. Noting $x = F^{-1}(F_X(x))$ and using the substitution $u = F_X(x)$ (with corresponding r.v. in upper case $U$), we have: $$\textrm{Var}(X_{(k)}) = \int_{0}^{1} \left(F_X^{-1}(u) - \mathbb{E}\left(F_X^{-1}(U)\right)\right)^2\,\mathcal{B}(u | k, n-k+1) \; \textrm{d}u,$$ where $\mathcal{B}$ denotes the PDF of a beta distribution (not the beta function). This can be rewritten as the variance based on the beta distribution: $$\textrm{Var}(X_{(k)}) = \textrm{Var}_{\mathcal{B}(u | k, n-k+1)}\left(F_X^{-1}(u)\right).$$ Approximating the RHS using (the variance bit of the) delta method, we get: $$\textrm{Var}_{\mathcal{B}(u | k, n-k+1)}\left(F_X^{-1}(u)\right) \approx \textrm{Var}_{\mathcal{B}(u | k, n-k+1)}(U) \cdot \left[(F_X^{-1})'\left(\mathbb{E}_{\mathcal{B}(u | k, n-k+1)}(U)\right)\right]^2,$$ where the prime denotes the derivative. The first term of the product is standard result. For the second part, given $(F^{-1})'(\cdot)=\frac{1}{f(F^{-1})(\cdot)}$ (see e.g. ), we then have: $$\textrm{Var}_{\mathcal{B}(u | k, n-k+1)}\left(F_X^{-1}(u)\right) \approx \frac{k(n-k+1)}{(n+1)^2(n+2)} \frac{1}{\left(f\left(F^{-1}\left(\frac{k}{n+1}\right)\right)\right)^2}.$$ Substituting in the standard normal distribution ($\phi$ for PDF, $\Phi$ for CDF) and scaling it to the desired variance $\sigma^2$ we arrive at: $$\textrm{Var}(X_{(k)}) \approx \frac{k(n-k+1)}{(n+1)^2(n+2)} \frac{1}{\left(\phi\left(\Phi^{-1}\left(\frac{k}{n+1}\right)\right)\right)^2},$$ which we can throw in a $\sigma^2$ term in front if it is not a standard normal. But surely someone must have derived that somewhere (again I am looking for some references), and there is a high chance that I have gone off track somewhere above.
11014
Let$ X_1, ...,X_n$ be a random sample feom a distribution $Geometric(\theta)$ for $0&lt;\theta&lt;1$. I.e,  $$p_{\theta}(x)=\theta(1-\theta)^{x-1} I_{\{1,2,...\}}(x)$$  Find the unbiased estimator with minimum variance for $g(\theta)=\frac{1}{\theta}$  My attempt:  Since the geometric distribution is from the exponential family, the statistics $$\sum X_i $$ is complete and sufficient for $ \theta$. Also, if $$T(X)=X_1$$ is an estimator for $g(\theta)$, it is unbiased. Therefore, by the Rao-Blackwell theorem and Lehmann-Scheffé Theorem, $$W(X) = E[X_1|\sum X_i]$$ is the estimator we are looking for.  We have the following:  $W(X) = \sum_{i=1}^t i\, P(X_1=i|\sum X_i =t) = \sum_{i=1}^t i\, \frac{P(\sum_{i \geq 2} X_i  =t-i)P(X_1=i)}{P(\sum_{i \geq 1}X_i =t)}$  Since the variables are iid geometric, the sums distributions are both negative binomials. But i am having troubles tosimplify the binomial coefficients and give a final answer with a better form, if it is possible.I wpuld be glad if I could get some help.  Thanks!  Edit: I dont think you guys understand my doubt: Ithink I made all the correct steps, maybe only forgot some indicator function. Here is what I did:  $$...=\sum_{i=1}^ti\frac{\binom{t-i-1}{n-2}\theta^{n-i}(1-\theta)^{t-i-n+1} \theta(1-\theta)^{i-1}}{\binom{t-1}{n-1}\theta^n(1-\theta)^{t-n}}=\sum_{i=1}^t i \frac{\binom{t-i-1}{n-2}}{\binom{t-1}{n-1}}$$  As i said, I am having troubles to simplify this and with the somatory index
118744
Suppose that $X$ ~ $Binomial(n,p)$ for $0 &lt; p &lt; 1$  Why does no unbiased estimator exist for $1/p$?  My approach:  We try to find the structure of $E_p(U(x))$, where $U(x)$ is any estimator of $1/p$.  Now, we will have:  $\sum{U(x)\binom{n}{x}p^x(1-p)^{n-x}}&lt;\sum{U(x)\binom{n}{x}}=M(n)&lt;\infty$  so that the expectation is bounded above. So this is supposed to mean that if $p &lt; 1/M(n)$,  then the expectation cannot attain $p$ but I am not sure why the above argument even makes sense and what being bounded means for the expectation.
53462
What is the variance of the sample variance? In other words I am looking for $\mathrm{Var}(S^2)$.  I have started by expanding out $\mathrm{Var}(S^2)$ into $E(S^4) - [E(S^2)]^2$  I know that $[E(S^2)]^2$ is $\sigma$ to the power of 4. And that is as far as I got.
117149
I am trying to do the following exercise:  Let $\xi_i,\dots, \xi_n$ be independent Bernoulli random variables such that $$ P\{\xi_i=0\}=1-\lambda_i \Delta\\ P\{\xi_i=1\}=\lambda_i \Delta $$ where $\Delta$ is a small number, $\Delta&gt;0,\lambda_i&gt;0$.  Show that $$ P\{\xi_1+\dots+\xi_n=1\}=(\sum_{i=1}^n\lambda_i)\Delta + O(\Delta^2) $$    I already wrote out (as already shown in part of @Marcin Malogrosz answer): $$ P\{\xi_1+\dots+\xi_n) = \sum_i \lambda_i \Delta \prod_{i\neq j} (1-\lambda_j\Delta) \\ \Bigl|\sum_i \lambda_i \Delta \prod_{i\neq j} (1-\lambda_j\Delta) - \sum_i \lambda_i \Delta\Bigr| = \sum_i \lambda_i \Delta \Bigl(1-\prod_{i\neq j} (1-\lambda_j\Delta) \Bigr) $$ But I am stuck here    I am able to show a simplified version with $P\{\xi_i=1\}=\lambda_i\Delta=\lambda\Delta$ (when the $n$ random variables are also identical) as following: $$ P\{\xi_1+\dots+\xi_n=1\}=\sum_i \lambda\Delta (1-\lambda\Delta)^{n-1} $$ using Taylor approximation when $\Delta$ is close to $0$ as $$ \lambda\Delta(1-\lambda\Delta)^{n-1}\Bigr|_{\Delta=0}+\frac{\Delta}{1!}\Bigl\{\lambda(1-\lambda\Delta)^{n-1}-\lambda^2\Delta^2(n-1)(1-\lambda\Delta)^{n-2}\Bigl\}\Bigr|_{\Delta=0} + O(\Delta^2)\\ =\lambda\Delta+O(\Delta^2) $$ and then sum over $i$.    My problem is to show:  $$ \Bigl(1-\prod_{i\neq j} (1-\lambda_j\Delta) \Bigr)=O(\Delta) $$  Thank you in advance
116793
I'm a mathematician self-studying statistics and struggling especially with the language.   In the book I'm using, there is the following problem:  A random variable $X$ is given as $\text{Pareto}(\alpha,60)$-distributed with $\alpha&gt;0$. (Of course, you could take any distribution depending on one parameter for the sake of this question.) Then a sample of five values $14$, $21$, $6$, $32$, $2$ is given.   First part: "Using the method of maximum likelihood, find an estimate $\hat{\alpha}$ of $\alpha$ based on [the sample]." This was no problem. The answer is $\hat{\alpha}\approx 4.6931$.  But then: "Give an estimate for the standard error of $\hat{\alpha}$."  What is meant by this? Since $\hat{\alpha}$ is just a fixed real number, I don't see in what way it could have a standard error. Am I to determine the standard deviation of $\text{Pareto}(\hat{\alpha},60)$?  If you think the question is not clear, this information would help me as well.
102687
Background  Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$  where $\mathbf{\beta}$ is an $(k\times1)$ vector of coefficients, $\mathbf{X}$ is the design matrix defined by  $$\mathbf{X} = \begin{pmatrix} 1 &amp; x_{11} &amp; x_{12} &amp; \dots &amp; x_{1\;(k-1)} \\ 1 &amp; x_{21} &amp; \dots  &amp;       &amp;   \vdots \\ \vdots &amp;   &amp; \ddots &amp;       &amp;   \vdots \\ 1 &amp; x_{n1} &amp; \dots  &amp; \dots &amp;   x_{n\;(k-1)}     \end{pmatrix}$$ and the errors are IID normal,  $$\mathbf{\epsilon} \sim \mathcal{N}\left(\mathbf{0},\sigma^2 \mathbf{I}\right) \;.$$  We minimize the sum-of-squared-errors by setting our estimates for $\mathbf{\beta}$ to be $$\mathbf{\hat{\beta}}= (\mathbf{X^T X})^{-1}\mathbf{X}^T \mathbf{y}\;. $$   An unbiased estimator of $\sigma^2$ is  $$s^2 = \frac{\left\Vert \mathbf{y}-\mathbf{\hat{y}}\right\Vert ^2}{n-p}$$ where $\mathbf{\hat{y}} \equiv \mathbf{X} \mathbf{\hat{\beta}}$ ().   The covariance of $\mathbf{\hat{\beta}}$ is given by $$\operatorname{Cov}\left(\mathbf{\hat{\beta}}\right) = \sigma^2 \mathbf{C}$$ where $\mathbf{C}\equiv(\mathbf{X}^T\mathbf{X})^{-1}$ () .  Question  How can I prove that for $\hat\beta_i$, $$\frac{\hat{\beta}_i - \beta_i} {s_{\hat{\beta}_i}} \sim t_{n-k}$$ where $t_{n-k}$ is a t-distribution with $(n-k)$ degrees of freedom, and the standard error of $\hat{\beta}_i$ is estimated by $s_{\hat{\beta}_i} = s\sqrt{c_{ii}}$.    My attempts  I know that for $n$ random variables sampled from $x\sim\mathcal{N}\left(\mu, \sigma^2\right)$, you can show that  $$\frac{\bar{x}-\mu}{s/\sqrt{n}} \sim t_{n-1} $$ by rewriting the LHS as $$\frac{ \left(\frac{\bar x - \mu}{\sigma/\sqrt{n}}\right)  }   {\sqrt{s^2/\sigma^2}}$$ and realizing that the numertor is a standard normal distribution, and the denominator is square root of a Chi-square distribution with df=(n-1) and divided by (n-1) ().  And therefore it follows a t-distribution with df=(n-1) ().  I was unable to extend this proof to my question...  Any ideas?  I'm aware of , but they don't explicitly prove it, they just give a rule of thumb, saying "each predictor costs you a degree of freedom".
7507
I am having an issue finding a best unbiased estimator for $\theta$. Any help is appreciated. Let $X_1, ..., X_n$ be a random sample from a population with pdf: $f(x\mid\theta)=\frac{1}{2\theta}$ $-\theta&lt;x&lt;\theta,\, \theta&gt;0$. I understand that $T(x) := |X_n|$ is the sufficient statistic since by the factorization theorem we have: $$f(x\mid \theta) = \left(\frac{1}{2\theta}\right)^n \prod_{i=1}^n I[|x_i|&lt;\theta]$$ I think my main issue is showing that this sufficient statistic is also a complete sufficient statistic. Can somebody please aid me in this?