The Durbin-Watson statistic to detect autocorrelation in the error terms ranges from 0 to 4. Currently, I am working out why it cannot exceed 4 analytically. The lower boundary case is obvious considering the actual statistic which is   $$DW =  \frac{\sum\limits_{t=2}^{T}(\hat{\epsilon}_t - \hat{\epsilon}_{t-1})^2}{\sum\limits_{t=1}^{T} \hat{\epsilon}_t^2};$$   but for the upper boundary case, i.e. DW &lt; 4, I ran in one particular difficulty. After expansion of the numerator I get   $$\sum\limits_{t=2}^{T}(\hat{\epsilon}_t - \hat{\epsilon}_{t-1})^2 = \sum\limits_{t=2}^{T} \hat{\epsilon}_t^2 - 2\hat{\epsilon}_t\hat{\epsilon}_{t-1} + \hat{\epsilon}_{t-1}^2 = \sum\limits_{t=2}^{T} \hat{\epsilon}_t^2 -2 \sum\limits_{t=2}^{T} \hat{\epsilon}_t\hat{\epsilon}_{t-1} + \sum\limits_{t=2}^{T} \hat{\epsilon}_{t-1}^2.$$   Obviously the last and first term are both less or equal than $\sum\limits_{t=1}^{T}\epsilon_t^2$ since they miss a non-negative term. Hence,   $$\sum\limits_{t=2}^{T} \hat{\epsilon}_t^2 + \sum\limits_{t=2}^{T} \hat{\epsilon}_{t-1}^2 \leq 2\sum\limits_{t=1}^{T} \hat{\epsilon}_t^2.$$   What I can't seem to figure out analytically is why necessarily   $$-2\sum\limits_{t=2}^{T}\hat{\epsilon}_t\hat{\epsilon}_{t-1} \leq 2 \sum\limits_{t=1}^{T} \hat{\epsilon}_t^2$$   must hold, i.e. why   $$0 \leq \sum\limits_{t=1}^{T} \hat{\epsilon}_t^2 + \sum\limits_{t=2}^{T} \hat{\epsilon}_t\hat{\epsilon}_{t-1},$$   considering that the error terms might be negative? Surely, there is some kind of estimation that yields the result but what is it?
113940
The test statistic for the Durbin Watson test can range from 0-4 from what I have gathered. Now the lower limit of 0 makes sense considering the test statistic consists of two summations which are both squared and divided by each other; but what gives us our upper limit of 4? Is this limit incorrect? Also, if the upper limit to this test statistic is not 4, than what is it?
87295
I'm interested in a good reference for results concerning asymptotic properties of maximum likelihood estimators. Consider a model $\{f_n(\cdot \mid \theta): \theta \in \Theta, n \in \mathbb N\}$ where $f_n(\mathbf x \mid \theta)$ is an $n$-dimensional density, and $\hat \theta_n$ is the MLE based on a sample $X_1, \ldots, X_n$ from $f_n(\cdot \mid \theta_0)$ where $\theta_0$ is the "true" value of $\theta$. There are two irregularities I'm interested in.    The data $X_1, \ldots, X_n$ are not iid and, as a result, the Fisher information about $\theta$ accrues at a rate slower than $n$.  $\Theta$ is a bounded set, and with positive probability $\hat \theta_n$ lies on the boundary. The boundary corresponds to a "simpler" model, and so there is particular interest in whether or not $\theta_0$ lies on the boundary.    My particular questions are   Letting $J_n(\theta)$ denote the observed Fisher information corresponding to $\theta$, and suppose $\theta_0$ lies in the interior of $\Theta$. Under what conditions is $$\left[J_n(\hat \theta_n)\right]^{1/2}(\hat \theta_n - \theta_0)$$ asymptotically normal as $n \to \infty$? In particular, are the regularity conditions similar to the usual ones, with the relevant modification being $J_n(\hat \theta_n) \to \infty$ in some sense? Suppose instead that $\theta_0$ is on the boundary, and again recall that $\hat \theta_n = \theta_0$ happens with positive probability - for concreteness, in a mixed effects model $Y_{ij} = \mu + \beta_i + \epsilon_{ij}$ we can have $\hat \sigma_{\beta}^2 = 0$. Under what conditions does $\hat \theta_n \to \theta_0$ (almost surely or in probability) and under what conditions will $\hat \theta_n = \theta_0$ eventually (this probably fails for the mixed effects model, but corresponds to "oracle" properties for the LASSO and related estimators, so perhaps it is too much to ask for general results)?   Again, just a pointer to a text with results at this level of generality would be greatly appreciated.
88502
Question  Suppose we have a multinomial distribution with $N$ possible outcomes, with probabilities $p_1,\ldots,p_N$. We sample this $n$ times, and denote the observed frequency of the $i$th outcome as $\xi_i$. In [1] the author claims that the distribution of the $\xi_i$ in the limit of large $n$ is:  $$p(\xi_1,\ldots,\xi_N)\propto\exp\left(-\frac{n}{2}\sum_{i=1}^N\frac{(\xi_i-p_i)^2}{p_i}\right).\;\;\;\;\;(1)$$  We can see immediately that this must be an approximation, as it assigns nonzero probabilities for $\xi_1+\cdots+\xi_N&gt;1$. However we can see that these have vanishing probability in the limit $n\rightarrow\infty$. My question is how do we derive (1) from the multinomial distribution, and show that they match in the $n\rightarrow\infty$ limit?  My thoughts  My first thought would be to appeal to the central limit theorem. The multinomial distribution has mean $\mu_i=p_i$ and covariance matrix $\Sigma_{ij}=\delta_{ij}p_i-p_ip_j$, so we would expect this in the large $n$ limit to be described by a multivariate Gaussian with mean $\mu$ and covariance $\frac{1}{n}\Sigma$. However, things are complicated by the fact that the multinomial covariance is singular (since $\xi_N$ is determined by the other $\xi_i$s), and so the multivariate Gaussian is not defined.  To address this, we may try and consider only the first $\xi_1,\ldots,\xi_{N-1}$, which have a non-singular covariance matrix and hence well-defined multivariate Gaussian distribution. Let's take the Binomial distribution $N=2$. The frequency $\xi_1$, this has mean $p_1$ and variance $p_1(1-p_1)$, so this would be described the the Gaussian: $$\propto\exp\left(-\frac{n}{2}\frac{(\xi_1-p_1)^2}{p_1(1-p_1)}\right).\;\;\;\;\;(2)$$ The expression (1) gives: $$\propto\exp\left(-\frac{n}{2}\left(\frac{(\xi_1-p_1)^2}{p_1}+\frac{(\xi_2-p_2)^2}{p_2}\right)\right).\;\;\;\;\;(3)$$ If we substitute $\xi_2\rightarrow 1-\xi_1$, $p_2\rightarrow 1-p_1$ into (3), we can verify that this gives the same answer as (2). I have verified that this also works for $N=4$.   I'm sure that if I just bashed out the algebra for general $N$ we would get agreement between the central limit theorem and (1) when we restrict the latter to $\xi_1+\cdots+\xi_N=1,p_1+\cdots+p_N=1$. However, how can we start with the multinomial distribution and derive (1) as a limit which is valid everywhere? One idea would be to say that (1) goes to zero as $n\rightarrow\infty$ when you are not on that plane, however I am a bit uncomfortable with this as it goes to zero everywhere except the mean as $n\rightarrow\infty$, so I don't know if that argument is good enough.  [1] Wootters, William K. "Statistical distance and Hilbert space." Physical Review D 23.2 (1981): 357.
54481
How is it that lowering the number of frequence,$n$ by $1$ in the for formula for Population and Sample Variance account for the discrepency of using sample rather than population.  I mean how is dividing the whole expression by $n-1$ in sample variance better than dividing by $n$?  I am referring to the formula $$s^2=\frac{\sum_{i=1}^{n}({x_i-x_{avg}})^2}{n-1}$$ more accurate than $$s^2=\frac{\sum_{i=1}^{n}({x_i-x_{avg}})^2}{n}$$  Thank You.
3016
This is an elementary question, but I wasn't able to find the answer.  I have two measurements: n1 events in time t1 and n2 events in time t2, both produced (say) by Poisson processes with possibly-different lambda values.  This is actually from a news article, which essentially claims that since $n_1/t_1\neq n_2/t_2$ that the two are different, but I'm not sure that the claim is valid.  Suppose that the time periods were not chosen maliciously (to maximize the events in one or the other).  Can I just do a t-test, or would that not be appropriate?  The number of events is too small for me to comfortably call the distributions approximately normal.
35720
I am reading the Pattern Recognition and Machine Learning book by Bishops. I have some difficulties proving the (6.21) equation. Can someone help me?   N.B: If someone has a better title for this question, i'm opened to it as i couldn't find the good words.
18406
The standard error of the intercept term ($\hat{\beta}_0$) in $y=\beta_1x+\beta_0+\varepsilon$ is given by $$SE(\hat{\beta}_0)^2 = \sigma^2\left[\frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right]$$ where $\bar{x}$ is the mean of the $x_i$'s.  From what I understand, the SE quantifies your uncertainty- for instance, in 95% of the samples, the interval $[\hat{\beta}_0-2SE,\hat{\beta}_0+2SE]$ will contain the true $\beta_0$. I fail to understand how the SE, a measure of uncertainty, increases with $\bar{x}$. If I simply shift my data, so that $\bar{x}=0$, my uncertainty goes down? That seems unreasonable.  An analogous interpretation is - in the uncentered version of my data, $\hat{\beta}_0$ corresponds to my prediction at $x=0$, while in the centered data, $\hat{\beta}_0$ corresponds to my prediction at $x=\bar{x}$. So does this then mean that my uncertainty about my prediction at $x=0$ is greater than my uncertainty about my prediction at $x=\bar{x}$? That seems unreasonable too, the error $\epsilon$ has the same variance for all values of $x$, so my uncertainty in my predicted values should be the same for all $x$.  There are gaps in my understanding I'm sure. Could somebody help me understand what's going on?
74314
I want to lower bound \begin{align}  \mathbb{P} \left( -\epsilon \leq  \sum_{i=1}^n X_i \leq 0 \right) &amp;= \mathbb{P} \left( - \frac{\epsilon}{\sqrt{\sum_{i=1}^n \sigma_i^2}} \leq  \frac{\sum_{i=1}^n X_i}{\sum_{i=1}^n \sigma_i^2} \leq 0 \right) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,(1) \end{align} where $\{X_i\}$ are mean zero, independent random variables with $\mathbb{E}[X_i^2] = \sigma_i^2 \leq H^2$ and $\mathbb{E}[|X_i|^3] = \lambda_i^3 \leq H^3$. From Equation $3$ in , I have that \begin{align} \sup_{x \in \mathbb{R}} |F_n(x) - \Phi(x)| \leq C_0 \left( \sum_{i=1}^n \sigma_i^2 \right)^{-3/2} \sum_{i=1}^n \lambda_i^3 \end{align} Loosening the bound above gives \begin{align} \sup_{x \in \mathbb{R}} |F_n(x) - \Phi(x)| \leq \frac{C_0}{\sqrt{n}} \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,(2) \end{align} Now I can write $(1)$ as \begin{align} F_n(0) - F_n \left(-  \frac{\epsilon}{\sqrt{\sum_{i=1}^n \sigma_i^2}} \right) \end{align} which by $(2)$ can be lower bounded as \begin{align} \Phi(0) - \frac{2C_0}{\sqrt{n}} - \Phi\left ( -  \frac{\epsilon}{\sqrt{\sum_{i=1}^n \sigma_i^2}} \right ) &amp;\geq \frac{1}{2} - \frac{2C_0}{\sqrt{n}} - \Phi\left ( -  \frac{\epsilon}{H \sqrt{n}} \right ) \end{align} Can the last expression be improved to something like $K/\sqrt{n}$ for some constant $K$ for large $n$? I feel like I need some approximations to normal CDF $\Phi(x)$ for small $x$.
83074
Question: Can anyone shed some light on this formula? I can't find any information on it.  It has three corollaries that I also need to understand:
49976
I've often seen the advice for checking whether or not a Poisson model fit is over-dispersed involving dividing the residual deviance by the degrees of freedom. The resulting ratio should be "approximately 1".  The question is what range are we talking about for "approximate" - what is a ratio that should set off alarms to go consider alternative model forms?
56137
There is a probabilistic method to solve it. But I am not familiar with probability. I am trying to compute it by analytic method, such as using L Hospital's rule or Stolz formula, but they are not working.
71275
In orthogonal design of lasso, we get  $\hat{\beta}_j^{\text{lasso}} = 0 \text{ if abs}(\hat{\beta}_j) \le \lambda /2$. WHY?   I've seen the answer and derived it myself, but don't know why.  We begin with definition of lasso, $$\hat{\beta}^{\text{lasso}} = \underset{x} {\arg\min}  \sum_{i=1}^{n} ( y_i - \sum_{j=1}^{p}\beta_{ij}x_{ij} )^2 + \lambda \sum_{j=1}^{p} |\beta_j|  $$  In orthogonal design case where $X^T X= I$, $\hat{\beta} = (X^TX)^{-1}X^{T}y = X^Ty$  \begin{align} L(\beta, \lambda) &amp; = \sum_{i=1}^{n} ( y_i - \sum_{j=1}^{p}\beta_{ij}x_{ij} )^2 + \lambda \sum_{j=1}^{p} |\beta_j| \\  &amp; = (Y - X \beta)^T(Y - X \beta) + \lambda \mathbf{I}_p \text{ abs}(\beta) \\  &amp; = Y^TY -2\hat{\beta}^T\beta + \beta^T \beta+ \lambda \mathbf{I}_p \text{ abs}(\beta) \\  &amp; = Y^TY + \sum_{j=1}^{p} L_j(\beta_j, \lambda) \end{align}  where $L_j(\beta_j, \lambda) = -2 \hat{\beta}_j \beta_j + 2\beta^2_j + \lambda \text{ abs}(\beta_j)$.  Leave aside $\beta_j=0$, take dereivative w.r.t. $\beta_j$ for abs$(\beta_j) &gt; 0$, $$\frac{L_j(\beta_j, \lambda)}{\partial \beta_j}  = -2 \hat{\beta}_j + 2\beta_j + \lambda \text{ sign}(\beta_j)$$  and $\hat{\beta}^{\text{lasso}}$ is either zero or solve, $$\beta_j + \lambda \text{ sign}(\beta_j) / 2 = \hat{\beta}_j,$$  which is, $$ \hat{\beta}^{lasso}_j =  \begin{cases}     \hat{\beta}_j - \lambda/2, &amp; \text{if } \hat{\beta}_j &gt; \lambda/2\\     \hat{\beta}_j + \lambda/2, &amp; \text{if } \hat{\beta}_j &lt; -\lambda/2 \end{cases} $$  My question is the following derivation,  If abs$(\hat{\beta}_j) \le \lambda / 2$, we get $$L_j(\beta, \lambda)  = -2 \hat{\beta}_j \beta_j + 2\beta^2_j + \lambda \text{ abs}(\beta_j)  \ge -\lambda \text{ abs}(\beta_j) + \lambda \text{ abs}(\beta_j) \ge 0 = L_j(0, \lambda)$$ and, we can tell $\hat{\beta}_j^{\text{lasso}} = 0 \text{ if abs}(\hat{\beta}_j) \le \lambda /2$ (Why? How can you tell?)  Why $\mathbf{\hat{\beta}_j^{\text{lasso}} = 0}$? The explanation of $L_j(\beta_j, \lambda) \ge L_j(0, \lambda)$ does not seem to justify the reason.
111648
Invariance property of MLE:  if $\hat{\theta}$ is the MLE of $\theta$, then for any function $f(\theta)$, the MLE of $f(\theta)$ is $f(\hat{\theta})$.    Also, $f$ must be a one-to-one function.  The book says, "For example, to estimate ${\theta}^2$, the square of a normal mean, the mapping is not one-to-one."  So, we can't use invariance property.  But then, it proves the property and says, "we now see that MLE of $\theta^2$, the square of a normal mean is $\bar{x}^2$".  This seems self-contradicting, we are squaring $\bar{x}$, but square of anything is not one-to-one, what am I reading wrong here?  Thanks!  source: Casella &amp; Berger "Statistical Inference"
37140
Does anyone have an intuitive explanation (no formulas, just words! :D) about the "$n-1$" instead of "$n$" in the unbiased variance estimator $$S_n^2 = \dfrac{\sum\limits_{i = 1}^n \left(X_i-\bar{X}\right)^2}{n-1}?$$
105675
I have 320 data points - each has a redshift and a turnover-frequency, and I want to fit a correlation between them (a linear fit). However, 120 of the turnover-frequency values are upper limits. As shown below the relationship is very weak:   the red points have upper limits for their turnover-frequency value. I would like to fit a straight line to this data that accounts for the upper limits. I think the tobit model in R can do this, but I'm not too sure (I'm new to R by the way); before I start with that, are there other methods in R I should consider?
51291
Let $\hat{\theta}_n= -\frac{n}{\sum_{i=1}^n \log(X_i)}$, where $X_i$ are i.i.d. samples from distribution with pdf $\theta x^{\theta-1}$ for $x \in (0,1)$. How to prove that $\hat{\theta}_n$ is consistent estimator of $\theta$.
53841
Given $m$ i.i.d. Bernoulli( $\theta$ ) r.v.s $X_{1}, X_{2}, \ldots, X_{m},$ I'm interested in finding the UMVUE of $(1-\theta)^{1/k}$, when $k$ is a positive integer. .  I know $\sum X_{i}$ is a sufficient statistic by the Factorization Theorem, but I'm having trouble proceeding from there. If I can find an unbiased function of the sufficient statistic the problem is solved by the Rao-Blackwell theorem.