In the 1972 paper Some Graphic and Semigraphic Displays, Tukey says that “[w]e all ought to be aware of $h^2 / 12$, the approximate increase in variance due to grouping in cells of width $h$. This reaches $2\%$ of the initial variance when $h=\sigma/2$, thus supporting the classical suggestion that frequency distributions with $10$ to $20$ occupied cells are adequate for most purposes”. I am uncertain of how these figures were derived. Preferably an answer would show the variance before and after grouping and then showing how the numbers above fit in.
6453
I have a dataset of sample observations, stored as counts within range bins. e.g.:  min/max  count 40/44    1 45/49    2 50/54    3 55/59    4 70/74    1   Now, finding an estimate of the average from this is pretty straight forward.  Simply use the mean (or median) of each range bin as the observation and the count as a weight and find the weighted average:  $$\bar{x}^* = \frac{1}{\sum_{i=1}^N w_i} \sum_{i=1}^N w_ix_i$$  For my test case, this gives me 53.82.  My question now is, what's the correct method of finding the standard deviation (or variance)?  Through my searching, I've found several answers, but I'm unsure which, if any, is actually appropriate for my dataset.  I was able to find the following formula both  and .  $$s^{2*} = \frac{ \sum_{i=1}^N w_i (x_i - \bar{x}^*)^2 }{ \frac{(M-1)}{M} \sum_{i=1}^N w_i }$$  Which gives a standard deviation of 8.35 for my test case.  However, the Wikipedia article on  gives both the formula:  $$s^{2*} = \frac{ \sum_{i=1}^N w_i}{(\sum_{i=1}^N w_i)^2 - \sum_{i=1}^N w_i^2} \sum_{i=1}^N w_i(x_i-\bar{x}^*)^2$$  and  $$s^{2*} = \frac{1}{(\sum_{i=1}^N w_i) - 1} \sum_{i=1}^N w_i(x_i-\bar{x}^*)^2$$  Which give standard deviations of 8.66 and 7.83, respectively, for my test case.  Update  Thanks to @whuber who suggested looking into Sheppard's Corrections, and your helpful comments related to them.  Unfortunately, I'm having a difficult time understanding the resources I can find about it (and I can't find any good examples).  To recap though, I understand that the following is a biased estimate of variance:  $$s^{2*} = \frac{1}{\sum_{i=1}^N w_i} \sum_{i=1}^N w_i(x_i-\bar{x}^*)^2$$  I also understand that most standard corrections for the bias are for direct random samples of a normal distribution.  Therefore, I see two potential issues for me:   These are binned random samples (which, I'm pretty sure, is where Sheppard's Corrections come in.)  It's unknown whether or not the data is for a normal distribution (thus I'm assuming not, which, I'm pretty sure, invalidates Sheppard's Corrections.)   So, my updated question is; What's the appropriate method for handling the bias imposed by the "simple" weighted standard deviation/variance formula on a non-normal distribution? Most specifically with regards to binned data.  Note: I'm using the following terms:     $s^{2*}$ is the weighted variance   $N$ is the number of observations.  (i.e. the number of bins) $M$ is the number of nonzero weights.  (i.e. the number of bins with counts) $w_i$ are the weights  (i.e. the counts) $x_i$ are the observations.  (i.e. the bin means) $\bar{x}^*$ is the weighted mean.
102059
How is the notation $X\sim N(\mu,\sigma^2)$ read? Is it $X$ follows a normal distribution? Or $X$ is a normal distribution? Or perhaps $X$ is approximately normal..  What if there are several variables that follow (or whatever the words is) the same distribution? How is it written?
1482
Is there a upper bound on the variance for a distribution that is confined say, in $[0,1]$, and has fixed mean $\mu \in [0, 1]$
89719
I'm confused as to the correct formula for approximate degrees of freedom to use for Welch's t-test. Satterthwaite's (1946) formula is the most commonly cited formula, but Welch gave an alternative in 1947. I'm not sure which is preferable (or used by most statistical software).  Satterthwaite's formula: $$\frac{\left(s_x^2/n_x +s_y^2/n_y\right)^2}{(s_x^2/n_x )^2/(n_x-1)+(s_y^2/n_y )^2/(n_y-1)}$$  Welch's formula: $$-2+ \frac{\left(s_x^2/n_x +s_y^2/n_y\right)^2}{(s_x^2/n_x )^2/(n_x+1)+(s_y^2/n_y )^2/(n_y+1)}$$  References:   Satterthwaite, F.E. (1946). "An Approximate Distribution of Estimates of Variance Components". Biometrics Bulletin, 2, 6, pp. 110–114.  Welch, B.L. (1947). "The generalization of "Student's" problem when several different population variances are involved". Biometrika, 34, 1/2, pp. 28–35.
6009
Recently, I have found in a  a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so.   Klammer, A. A., Park, C. Y., and Stafford Noble, W. (2009) . Journal of Proteome Research. 8(4): 2106–2113.
116875
let $Z \sim N(0,1)$ and $W \sim \chi^2(s)$.  If $Z$ and $W$ are independently distributed then the variable $Y = \frac{Z}{\sqrt{W/s}}$ follows a $t$ distribution with degrees of freedom $s$.  I am looking for a proof of this fact, a reference is good enough if you do not want to write down the complete argument.
37140
Does anyone have an intuitive explanation (no formulas, just words! :D) about the "$n-1$" instead of "$n$" in the unbiased variance estimator $$S_n^2 = \dfrac{\sum\limits_{i = 1}^n \left(X_i-\bar{X}\right)^2}{n-1}?$$
19487
The number of different ways to choose $k$ out of $n$ unique items:   Without repetition and without order-significance: $\binom{n}{k}=\frac{n!}{k!(n-k)!}$ Without repetition and with order-significance: $\binom{n}{k}\times k!=\frac{n!}{(n-k)!}$ With repetition and with order-significance: $n^k$ With repetition and without order-significance: $?$   Thank you!
102707
I would like to test $H_{0} :$ $\mu_{1} = \mu_{2}, \ \sigma_{1}^{2} = \sigma_{2}^{2}$ vs $H_{0} :$ $\mu_{1} \neq \mu_{2}, \ \sigma_{1}^{2} \neq \sigma_{2}^{2}$Is there a test statistic that would allow me to detect both differences in the mean and differences in the variance? Is there a way that I could possibly combine the $F$ - $test$ and the $t$ - $test$ to derive this test statistic? I am assuming that the population is normally distributed but what if it wasn't?I am very stuck so any help would be greatly appreciated.
98070
I'm trying to understand the concept of capacity for a wireless channel.  Some help would be appreciated.  For a AWGN channel capacity is calculated as:  $$C=B \cdot log_2(1 + S/N)\text{ bits/sec}$$  B = bandwidth.  This is what I don't understand.  Why isn't it a factor of frequency? To me considering bandwidth only makes sense in cases where the system changes frequency.   Bandwidth is the difference between an upper and lower frequency range.  Well, what if I'm using a fixed-frequency signal?  Fupper and Flower would be the same value, right?  So does that mean B=0?  So a fixed frequency signal can't carry any data?  We know that's not true, AM radio does it.  So what am I missing? According to this formula, a fixed-frequency signal would have the same performance regardless of whether it's at high or low frequency.  This makes no sense to me.  For example say my bandwidth is 1Hz at a fixed frequency of 1Hz.  Compare this with a bandwidth of 1Hz at a frequency of 2.4GHz.  It's plainly obvious that I can cram way more bits into 2.4 x 109 cycles/second than I can with just 1/sec.  But according to this formula I can't.  Please help. What about fractional differences?  Waveforms are analog in nature, so we could have a 1Hz signal and a 1.5Hz signal.  Likewise at the high frequency range.  Say 2.4GHz minus 0.5Hz.  There is an infinite amount of space between 1 and 1.5.  Could not 1Hz and 1.001Hz serve as two separate channels?  In terms of practicality I realize this would be difficult, nearly impossible to measure this difference with modern electronics, especially with noise added, but in pure theory you could have two channels.  So in that sense, shouldn't there be an infinite amount of bandwidth between two frequencies?  Or do we only count in 1Hz whole number increments?
50064
So there is  about  why variance  is squared.   And the answer seems to be "because we get to do groovy maths when it is squared". Ok, that's cool, I can dig.  However, I'm sitting reading some financial maths stuff, and a lot of the equations on pricing and risk are based on variance. It doesn't seem to be the best basis for, you know, pricing exotic vehicles that are worth in the millions (or billions) that a formula for variance is used "because the maths is better".    To make a point  then, why not have the variance be from the cubed, abs cubed, or the 4th power (or even a negative power)?   eg (apologies, I don't know Latex)  Sum 1/N * |(x - mean)^3|   OR  Sum 1/N * (x - mean)^4   Would using variance-to-a-different power measurably alter pricings/valuations if the equations still used variance as usual (but the variance was calculated with the different power)?  Is there a reason why we stopped  at "power of 2", and are there any implications of using a variance concocted from a different (higher or lower) power?
85678
I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.   If we use a less common method like LAD, where we just consider the absolute deviation, then outliers make less difference to the final model, while if we take the cube of the error (or any other power higher than 2), then outliers are far more significant than with the least squares model.  I suppose what I'm really asking is mathematically, is raising the error to the power of 2 really that special. Is it say more "accurate" in some sense than raising the error to the power of 1.95 or 2.05???   Thanks!
95976
Unbiased weighted variance was already addressed  and  but there still seems to be a surprising amount of confusion.  There appears to be a consensus toward the formula presented in the first link as well as in the .  This also looks like the formula used by R, Mathematica, and GSL (but not MATLAB).  However, the Wikipedia article also contains the following line which looks like a great sanity check for a weighted variance implementation:     For example, if values {2,2,4,5,5,5} are drawn from the same   distribution, then we can treat this set as an unweighted sample, or   we can treat it as the weighted sample {2,4,5} with corresponding   weights {2,1,3}, and we should get the same results.   My calculations give the value of 2.1667 for variance of the original values and 2.9545 for the weighted variance.  Should I really expect them to be the same?  Why or why not?
90068
I wish to better understand how the  to the binomial distribution for the normal approximation was derived.  What method was used to decide we should add 1/2 (why not another number?). Any explanation (or a link to suggested reading, other than , would be appreciated).
59606
I am creating a component that aims to calculate the average and variance of a metric associated with events happening during time but with a limited internal memory.  Imagine that the events are visitors entering in a shop and the metric is their age.  During time, my component receives events with the age of each visitor. I don't want my component to memorize the history of each ages. Ideally, I would like a light component storing only: the average A, the variance V and the number of events N.  After each event with age E, I want to update those three values :  N&lt;=N+1 A&lt;=(A*N+E)/(N+1) V&lt;=???   What for V? I am thinking of something like :  V&lt;=(V*N+(E-A)^2)/(N+1)   I know it is not exact as my previous V is using the old A which is no more the average.   Q1 - Is there an exact formula? Q2 - If not, is my proposal a good estimate? Is it biased? Will it converge correctly when N increases? Q3 - Is there a better formula?
72516
Having read a little about exchangeability, I went back to thinking about the iid condition required for the central limit theorem. It struck me that if two random variables are drawn from an identical distribution, the occurrence of one event does not make it more or less probable for the occurrence of another event. I have no doubts that I am wrong and there is a reason why both independence and identical distribution are required. I just don't know why!
8875
I understand that when sampling from a finite population and our sample size is more than 5% of the population, we need to make a correction on the sample's mean and standard error using this formula:  $\hspace{10mm} FPC=\sqrt{\frac{N-n}{N-1}}$  Where $N$ is the population size and $n$ is the sample size.  I have 3 questions about this formula:   Why is the threshold set at 5%? How was the formula derived? Are there other online resources that comprehensively explain this formula besides  paper?