If I wanted to maximum likelihood estimator for $f(x)=\frac{1}{\sqrt{\sigma^22\pi}}e^{-\frac{1}{2}(\frac{x-\mu}{\sigma})^2}$ where $-\infty\leq x \leq \infty$, my original plan was to do $\Pi^{n=\infty}f(x)$ but that really doesn't make any sense. The confusion arises since when you usually find the MLE, we' do $\Pi^{n}_{i=1}f(x)$ but I'm given the bound $-\infty\leq x \leq \infty$ so I can't just simply do the old trick of doing $\ln(L(\theta)) = ...$ and taking the derivative since there's going to be INFINITE terms Other similar questions asked on this stackExchange has been bounded by some $n$ but what if it isn't?
121545
In MLE for continuous case, my course notes define the likelihood function to be:  $$ L(\theta) = L(\theta;y) = \prod_{i=1}^n f(y_i;\theta) $$  Where $f$ is the joint pdf of $y_i$ given $\theta$.  I thought normally it doesn't make sense to evaluate a pdf at a point (eg. $f(X=2) = 0$ for some continuous rv X), and that it only make sense to evaluate for the random variable to be $&gt;$ or $&lt;$ to a value.  But in the case of MLE, why is it OK to evaluate a joint pdf at a point?
112658
Is it enough to show that MSE = 0 as $n\rightarrow\infty$? I also read in my notes something about plim. How do I find plim and use it to show that the estimator is consistent?
96005
Let $X_1,\ldots,X_n$ be i.i.d. $B(1,\theta)$ random variables, $0&lt;\theta&lt;1$. Then, as an estimator $\theta$, check if $T(X_1,\ldots,X_n)= \dfrac{\sum_{i=1}^n X_i +\sqrt{n}/2}{n+\sqrt{n}}$ is consistent and/or unbiased.  $$T=\frac{\frac{1}{n}\sum_{i=1}^n X_i +\frac{1}{2\sqrt{n}}}{1+\frac{1}{\sqrt{n}}}$$  $$T=\frac{\bar{X} +\frac{1}{2\sqrt{n}}}{1+\frac{1}{\sqrt{n}}}$$  $$E(T)=\frac{E(\bar{X} +\frac{1}{2\sqrt{n}})}{1+\frac{1}{\sqrt{n}}}=\dfrac{\theta +\frac{1}{2\sqrt{n}}}{1+\frac{1}{\sqrt{n}}}.$$  So, $T$ is biased.  Consistency: $\lim\limits_{n \to\infty}E(T)=\lim\limits_{n \to\infty}\dfrac{E(\theta +\frac{1}{2\sqrt{n}})}{1+\frac{1}{\sqrt{n}}}$  So $T$ is consistent.   But The given answer is "neither unbiased nor consistent"  Where did I go wrong ? Please advise.
45520
I'm taking a Mathematical Statistics course and trying to work through a homework problem that reads:  Let Y1, ..., Yn be a random sample from a Beta(1,$\theta$) population.  Derive the MLE for $\theta$.  Getting started, I set up the likelihood function as:  $$ L(\theta) = \frac{\Gamma(1+\theta)}{\Gamma(1)\Gamma(\theta)}\prod_{i=1}^n(1-Y_i)^{\theta-1} $$  This simplifies to:  $$ L(\theta) = \theta(1-Y_i)^{n(\theta-1)} $$  Taking the log:  $$ l(L) = log(\theta) + n(\theta-1)log(1-Y_i) $$  Take the derivative:  $$ \frac{dl}{d\theta} = \frac{1}{\theta}+nlog(1-Y_i) $$  Solve for $\theta$:  $$ \theta = \frac{-1}{nlog(1-Y_i)} $$  But this can't be right as I've not dealt with the summation of $Y_i$.  In most examples I've seen, this goes away as a result of the summation being divided by n,b ut in this case I can't find where I've gone wrong.   I appreciate any tips.
109327
Suppose that $U_n=\frac{1}{\sqrt{2n\sigma^2}}\left(\Sigma X_j-\Sigma Y_j\right)$,  where $X_1,X_2,\ldots$ and $Y_i,Y_2, \ldots$ are i.i.d. sequences of random variables with mean $\mu$ and variance $\sigma^2$ and the sequences $\{X_i\}$ and $\{Y_i\}$ are independent.  Then, for all real numbers $x$, show that, as $n\to \infty$,  $$\mathbb{P}(U_n\leq x) \to  \int_{-\infty}^{x} \frac{1}{(\sqrt(2\pi))} \:e^{-\frac 12{u^2}}\mathrm du $$ Approach  I tried to carry out the integration on the right, but this does not integrate nicely. It ends up involving an error function, so I figured this was the wrong was to go.  Should I be looking at convergence in probability to answer this question? Is Chebyshev's inequality relevant here? If so, how and how does that relate to the integral?
59338
Let $f\in L^{2} (\mathbb R)$ with $\lim_{t\to \pm \infty} f(t)=0.$  Put, $F_{n} (x)= \frac{1}{2\pi} \int_{-n}^{n}e^{itx} f(t) dt, \ (n=1,2,...).$ Fix $\alpha \in (0, \infty)$ and we define $H_{n}(x)$ as follows:  $$\frac{1}{2\pi}\int_{-n}^{n} e^{itx} (f(t+\alpha) -f(t-\alpha))dt = (e^{-i\alpha x}- e^{i\alpha x})F_{n}(x) + H_{n}(x).$$     My Question: How to show $H_{n}(x) \to 0$ as $n\to \infty$ in $L^{2}(\mathbb R)$ ?   (I guess some where we need to use ; but I am bit confused, how to use it.)   Edit: @, pointed out below, and in MO, for the same,  I have tried little in that direction:  $$H_{n}(x)= \frac{1}{2\pi}\int_{-n}^{n} e^{itx} (f(t+\alpha) -f(t-\alpha))dt -(e^{-i\alpha x}- e^{i\alpha x}) \frac{1}{2\pi} \int_{-n}^{n}e^{itx} f(t) dt$$ we may write, $H_{n}(x)= \frac{1}{2\pi} e^{-i\alpha x} \int_{-n+\alpha}^{n+\alpha}e^{itx} f(t) dt- \frac{1}{2\pi} e^{i\alpha x} \int_{-n-\alpha}^{n-\alpha} e^{itx} f(t) dt-(e^{-i\alpha x}- e^{i\alpha x}) \frac{1}{2\pi} \int_{-n}^{n}e^{itx} f(t) dt;$ Or, we may write,  $H_{n}(x)= \frac{1}{2\pi} e^{-i\alpha x} \int_{-n+\alpha}^{n+\alpha}e^{itx} f(t) dt- \frac{1}{2\pi} e^{i\alpha x} \int_{-n-\alpha}^{n-\alpha} e^{itx} f(t) dt- \frac{1}{2\pi}\int_{-n-\alpha}^{n+\alpha} e^{itx} f(t+\alpha) dt + \int_{-n+\alpha}^{n+\alpha} e^{itx} f(t-\alpha) dt= I_{1}-I_{2}-I_{3}+I_{4};$  But then,(using one of the above form), we need to show:  $\lim_{n\to \infty}\int_{\mathbb R} |H_{n}(x)|^{2} dx=0.$  Now If I interchange the limit and integral (this may not be possible, here, I am doing without rigours), and then I can take limit inside the square, and then after taking limit, each integrals, $I_{i}, (i=1,2,3,4)$, will be becomes over $\mathbb R$; then I think, it will cancels each other;      Is this o.k or am I missing something, still, I don't see, where we need to use asymptotic behaviours of $f$ (and what does it mean ?, here is $f$ is arbitrary member of $L^{2}$) ? I am wondering, is it possible to get solution without, Plancherel or parsevel somewhere ?
42104
Let $X_1, X_2, \ldots , X_n$ be a random sample from a distribution with the following pdf  $$f(x|\theta) =  \begin{cases} 1/(\theta_2−\theta_1), &amp;\quad\text {for}\quad \theta_1 \leq x\leq \theta_2\\ 0 &amp;\quad\text { otherwise}\quad \end{cases}$$ Suppose that $\theta_1$ and $\theta_2$ are unknown.  How would I go about writing the likelihood function for this distribution on $\theta_1$ and $\theta_2$.
57133
In a statistics book i'm reading, it is postulated that asymptotic normality of an estimator implies consistency. That is $$ \hat{\theta}_n \stackrel{as}{\sim} \mathcal{N}(\theta_0, \frac{1}{n}\sigma(\theta_0)) \Rightarrow P_{\theta_0}(|\hat{\theta}_n - \theta_0|&gt;\epsilon ) \to 0 $$ when $n\to\infty$, for all $\theta_0 \in \Theta$ and $\epsilon&gt;0$.  I am trying to prove this, but i can't seem to get a breakthrough.   If anyone could shed some light on how this is proven, i would be very grateful.
28779
Let $X_1, ..., X_n$ be iid. and $X_1\sim N(\mu,1)$. $\gamma(\mu)=e^{t\mu}$ for $t\neq 0$    My question is how to find an UMVU estimator for $\gamma(\mu)$  My concern is not so much about the specific problem but rather about how to approach this  kind of estimation problem in general, if one wants to estimate a non-linear transformation of a parameter. The problem for me is that $\overline{X}_n$ is UMVUE for $\mu$ but I think that: $E_\theta[e^{\overline{X}_n}]\neq e^{t\theta}$.   We never covered this kind of problem in our lecture so I dont really have an idea of how to solve this kind of estimation problem  EDIT: I think I solved it myself: Let $\frac{1}{k}:=\frac{1}{(2\pi)^{n/2}}$  $ \int e^{t\overline{X}_n} \cdot \frac{1}{k}\exp(-\frac{1}{2}\sum_{i=1}^n(x_i-\theta)^2)d(x_1, ..., x_n)= $  $=\int \frac{1}{k}\exp(-\frac{1}{2}\sum_{i=1}^n(x_i-2(\theta+\frac{t}{n})x_i+\theta^2)d(x_1, ..., x_n)= $  $=\exp(t\theta+\frac{t^2}{2n})\int \frac{1}{k}\exp(-\frac{1}{2}\sum_{i=1}^n(x_i-2(\theta+\frac{t}{n})x_i+(\theta+\frac{t}{n})^2)d(x_1, ..., x_n)=$  $=\exp(t\theta+\frac{t^2}{2n})$  The UMVU estimator is therefore $T(X1, ..., X_n)=\exp(-t^2/2n) \exp(t\overline{X}_n)$ It is UMVU because $\overline{X}_n $ is sufficient and thereby by Lehmann-Scheffe the estimator is UMVU, as $T(X_1, ... X_n)=g(\overline{X}_n)$  Remark: I would still have to proof that $Var_\theta(g(\overline{X}_n)) &lt; \infty \forall \theta $ for Lehmann-Scheffe to be applicable.
48669
Let $X_1,X_2,\dots, X_n$ be a random sample form a distribution $f(x)=e^{\delta-x}$ for $x\geq\delta$. Find MLE of $\delta$.   My solution:  $$L(\hat\gamma)=\prod_{i=1}^nf(x_i\mid \delta)=\prod_{i=1}^ne^{\delta-x} =e^{n\delta}e^{\sum_{i=1}^nx_i}$$  The log likelihood is $l(\hat\delta)=n\delta-\sum_{i=1}^n x_i$. To find the $\max$ we take the 1st derivative with respect to $\delta$: $$l'(\hat\delta)=n$$ the first derivative cannot be zero. I am not sure how to proceed.
59756
Say we have $f(x , \theta) = \frac{1}{2}e^{-|x-\theta|}$ Lets assume for simplicity, we only have 1 sample. We find that the log-likelihood for this distribution is:  $$ l(\theta , x) = -log(2) + (\theta - x) I_{(x &lt; \theta)}+(x-\theta)I_{(x \geq \theta)}. $$  Now in general, Call the score function:  $$ S(\theta , X) = Dl = \frac{Dg_{\theta}}{g_{\theta}} $$ by noticing the log-likelihood is expressed as :  $$ l(\theta, X) = log (g_{\theta}) $$  Where g is the likelihood and assume $g$ is 3 times differentiable. Furthermore, we find :  $$ DS = \frac{D^2g_\theta}{g_\theta}- SS'. $$ Which finally allows us to obtain that :  $$ var_\theta S = -E_\theta DS $$ by noticing that $E_\theta S = 0$  Now in this case:  $$ Dl = \frac{dl}{d\theta} = I_{(x &lt; \theta)} - I_{(x \geq \theta)} $$  So indeed  $$ E_\theta S = P{(x &lt; \theta)} - P{(x \geq \theta)} = 0 $$ And then $\frac{dS}{d\theta} = 0$ so $-E_\theta \frac{dS}{d\theta} = 0$  Now computing  $$ ES^2 = E( [I_{(x &lt; \theta)} - I_{(x \geq \theta)}]^2) = E(I_{(x &lt; \theta)} + I_{(x \geq \theta)}) = 1. $$ This does not match the case above, however here do not have differentiability. Is what I did correct, or plain fiction?
42019
Exercise :     Calculate a Maximum Likelihood Estimator for the model $X_1,\dots, X_n \; \sim U(-\theta,\theta)$.   Solution :  The distribution function $f(x)$ for the given Uniform model is :  $$f(x) = \begin{cases} 1/2\theta, \; \; -\theta \leq x \leq \theta \\ 0 \quad \; \; , \quad\text{elsewhere} \end{cases}$$  Thus, we can calculate the likelihood function as :  $$L(\theta)=\bigg(\frac{1}{2\theta}\bigg)^n\prod_{i=1}^n\mathbb I_{[-\theta,\theta]}(x_i)= \bigg(\frac{1}{2\theta}\bigg)^n\prod_{i=1}^n \mathbb I_{[0,\theta]}(|x_i|) $$  $$=$$  $$\bigg(\frac{1}{2\theta}\bigg)^n\prod_{i=1}^n \mathbb I_{[-\infty,\theta]}(|x_i|)\prod_{i=1}^n \mathbb I_{[0, +\infty]}(|x_i|)$$  $$=$$  $$\boxed{\bigg(\frac{1}{2\theta}\bigg)^n\prod_{i=1}^n \mathbb I_{[-\infty,\theta]}(\max|x_i|)}$$  Question : How does one derive the final expression in the box from the previous one ? I can't seem to comprehend how this is equal to the step before.  Other than that, to find the maximum likelihood estimator you need a $\theta$ sufficiently small but also $\max |x_i| \leq \theta$ which means that the MLE is : $\hat{\theta} = \max |x_i|$.
62306
What would the normalising  constant be of the following, or atleast an approximation? I would like to avoid sampling.  $$f(\theta)=\exp(-k_1e^{-k_2\theta^2}-\theta^2)\qquad\theta\in(-\infty,\infty), \quad k_1,k_2&gt;0$$  In the event that its not possible, what would be a valid polynomial lower bound on $\log(f(\theta))$, that would take into account $k_1$  For anyone, who is interested, I end up with this expresssion because I place a Gaussian prior over (a squared exponential) kernel parameters, and trying to use variational bayes (hence, a lower bound).
78779
Peace be upon you,  At the end of this question, I have shown that how computing MLE on an i.i.d Beta distributed data, results in the following system \begin{align*} &amp;\begin{cases} \psi(\alpha)-\psi(\alpha+\beta) = c_1\\ \psi(\beta)-\psi(\alpha+\beta) = c_2 \end{cases}\\ &amp;c_1 = \frac{1}{n} \left[ \sum_{i=1}^n {log(x_i)}\right] &amp;c_2 = \frac{1}{n} \left[ \sum_{i=1}^n {log(1-x_i)} \right] \end{align*} where $\psi(.)$ is the digamma function and $x_i\in(0,1]$.  Since I have about 19k of the above system to be solved, then using the iterative methods like Newton-Raphson would take a long time. So, I would like to find a fast-computable late-diverging approximating answer for it. I followed the idea of estimating the $\psi(.)$ function by a, likely, invertible function and substituting it and then solving a new system which results in a closed-form approximating solution.  The simplest candidate for this idea is the Taylor expansion; but it would have a very bad accuracy by getting far from the origin or in other words it diverges quickly.  I suppose that for some functions the divergence would occur very late. For example, $\ln(x)$ is very similar to $ψ(x)$ and it may keep the precision in a large domain. but for such suggestion there is another problem: how to solve the system after substitution.  Maybe, someone will suggest another way for solving the above system. But if some user wants to solve it by the aforementioned technique, he/she can find a much similar basis (than $\{1,x,x^2,...\}$) for a $ψ(x)$ estimation which makes the above system has a closed-form answer.  Note that I need a "late-diverging" fast-computable answer, either a closed-form approximating answer or any other fast-computable one.    But, how did I reach this system?  We know that the probability density function of the Beta distribution is \begin{align*} f(x|\alpha, \beta)=\frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)} \end{align*} So, the likelihood is \begin{align*} \mathcal{L}(\alpha,\beta)=f(x_1,x_2,...,x_n|\alpha,\beta)=\prod_{i=1}^n{f(x_i|\alpha,\beta)} \end{align*} For pre-simplifying the computations, we proceed to find the maximum of the log-likelihood \begin{align*} log(\mathcal{L}(\alpha,\beta))=\sum_{i=1}^n{log\left(f(x_i|\alpha,\beta)\right)} \end{align*} For maximizing the above log-likelihood, we should solve the following system \begin{align*} \begin{cases} \frac{\partial}{\partial \alpha}\left[ \sum_{i=1}^n {(\alpha-1)log(x_i)+(\beta-1)log(1-x_i)-log(B(\alpha,\beta))} \right]=0\\ \frac{\partial}{\partial \alpha}\left[ \sum_{i=1}^n {(\alpha-1)log(x_i)+(\beta-1)log(1-x_i)-log(B(\alpha,\beta))} \right]=0 \end{cases} \end{align*} So, we can write \begin{align*} \begin{cases} \left[ \sum_{i=1}^n {log(x_i)}\right] =n\frac{\partial}{\partial\alpha}\left[log(B(\alpha,\beta)\right]\\ \left[ \sum_{i=1}^n {log(1-x_i)} \right]=n\frac{\partial}{\partial\beta}\left[log(B(\alpha,\beta)\right] \end{cases} \end{align*} Using the formula for partial derivation of Beta function we can write \begin{align*} \begin{cases} \left[ \sum_{i=1}^n {log(x_i)}\right] =n\left[\psi(\alpha)-\psi(\alpha+\beta)\right]\\ \left[ \sum_{i=1}^n {log(1-x_i)} \right]=n\left[\psi(\beta)-\psi(\alpha+\beta)\right] \end{cases} \end{align*} where $\psi(.)$ is the digamma function. we can rewrite the above system as \begin{align*} &amp;\begin{cases} \psi(\alpha)-\psi(\alpha+\beta) = c_1\\ \psi(\beta)-\psi(\alpha+\beta) = c_2 \end{cases}\\ &amp;c_1 = \frac{1}{n} \left[ \sum_{i=1}^n {log(x_i)}\right] &amp;c_2 = \frac{1}{n} \left[ \sum_{i=1}^n {log(1-x_i)} \right] \end{align*}
28926
Let $X_1,...X_n$ be a random sample from the Pareto distribution with parameters $\alpha$ and $\theta$, where $\alpha$ is known.   Find the maximum likelihood estimator for $\theta$ and say if it is unbiased, if not find an unbiased estimator  My Approach:  $$f(x;\alpha, \theta) = \alpha \theta^\alpha x^{-(\alpha +1)},\quad x \ge \beta$$  $$L(\theta) = \alpha^n \theta^{\alpha n} \left(\prod_{i=1}^n x_i\right)^{-(\alpha+1)}$$  Taking log for $L(\alpha)$ gives  $$\ln L(\theta) = n \ln(\alpha) + \alpha n \ln(\theta) + \sum_{i=1}^n -(\alpha+1) \ln(x_i)$$  Then since $\ln L(\theta)$ is an increasing function if $\theta$ increases, and for a Pareto distribution we have that $\theta \le x$ we conclude that the maximum likelihood estimator is $\hat\theta=\min {x_i}$ (the first order statistic) Am I right?  Then to prove that it is and unbised statistic we have to prove that $E(\hat{\theta}) = \theta$.  I donot know how to do it I just thought using the p.d.f of the firs order statistic and integrate from $\theta $ to infinite but I'm not sure about this.  Any ideas?
26389
Maximum of a Sample from a Uniform Distribution:  Suppose $X_1, ...  , X_n$ is a random sample for a $\mathrm{uniform}(0,\theta)$ distribution. Suppose $\theta$ is unknown. An intuitive estimate of $\theta$ is the maximum of a sample. Let $Y_n = \max\{X_1, ... , X_n\}$. Exercise 5.1.4 shows that the cdf of $Y_n$ is   $F_{Y_n}(t) = 1$ if $t&gt;\theta$, $F_{Y_n}(t) = \frac{t^n}{\theta^n}$ if $0 &lt; t \leq \theta$, and $F_{Y_n}(t) = 0$ if $t\le0$.   Then the pdf of $Y_n$ is $f_{Y_n}(t) = \frac{nt^{n-1}}{\theta^n}$ if $0 &lt; t \leq \theta$, and $f_{Y_n}(t) = 0$ elsewhere.  Based on its pdf, it is easy to show that $E(Y_n) = (n/(n+1))\theta$. Thus, $Y_n$ is a biased estimator $\theta$... Further, based on the cdf of $Y_n$, it is easily seen that $Y_n$ converges to $\theta$ in probability.   MY QUESTION:     How do we know that $Y_n$ converges to $\theta$ in probability? Is it because $E(Y_n) \rightarrow \theta$?   Thanks in advance.
28918
I proceeded by finding $\operatorname E(\bar X).$ I considered $\bar X$ as a constant and simply got the term itself. This should suggest that $\bar X$ is not an unbiased estimator of $\theta$. That's obvious too cuz it's a constant(from what I assumed). However, I'm also aware that $\mu = \theta$ so that's probably not right.  I have to demonstrate first that $\bar X$ is an unbiased estimator of $\mu$.