I have a sample with data and I would like to compare the result I get from Monte Carlo integration and by analytically computing the marginal likelihood. Now I know that in many cases this is a challenging computation, but in this case apparently it is possible. I have the code for doing it, but I don't understand how it is done, and the code doesn't make me wiser.  So, let  $Y \sim N(\theta,1)$ and the prior on $\theta \sim N(0,1)$.  How would I analytically compute the  marginal likelihood $p(y)$  $$p(y) = \int p(y|\theta)\,p(\theta)\, d\theta$$
123566
Given that $a$~$\mathcal{N}(\mu, \Sigma)$ and $\mu$~$\mathcal{N}(0, \sigma^2 I)$, Is $p(a|\Sigma, \sigma)=\int p(a|\mu, \Sigma)p(\mu|\sigma)d\mu$ again a Normal distribution? And how to estimate the mean and variance respectively? Thanks a lot!
1465
I have 2 i.i.d Uniform random variables $\theta_{1}$ and $\theta_{2}$ on $[0, 2\pi)$. Here the problem pushes us to consider that all operations are made modulo $2\pi$. My question is how to obtain the joint PDF of $\theta_{1} - \theta_{2}$ and $\theta_{1}$ and then obtain the conditional PDF of $\theta_{1}$ given $\theta_{1} - \theta_{2}$??. For the second part It can be deduced from the first by writing it as the joint PDF divided by the PDF of $\theta_{1}$. But for the first part, A common idea that I saw is to derive the $\text{CDF}(x, y) = P(\theta_{1} - \theta_{2} \le x$, $\theta_{1} \le y$) and then perform derivations respect to $x$ and $y$. Is this a right approach? If yes, how to proceed with CDF especially with our uniform distribution restrictions?
79868
Suppose, we have $n$ samples $x_i$ of a random variable: $$x \sim \mathcal N(\mu,\sigma^2) $$ Based on the samples, we want to estimate the probability that $x$ is negative: $$P(x\le0)$$ Intuitively, I would first estimate: $$\hat \mu=\frac 1 n \sum x_i$$ $$\hat \sigma^2={\frac 1 {n-1} \sum (x_i-\hat \mu)^2}$$ and then calculate: $$P(x\le0)=\frac1{\sqrt{2\pi\hat\sigma^2}} \int_{-\infty}^0 e^\frac{x-\hat\mu}{\hat \sigma} dx$$ However $\hat \mu$ and $\hat \sigma$ have variance! If I use this method, I suspect I am ignoring that variance and making an incorrect estimation. Is this reasoning right? If so, how can I estimate $P(x\le0)$ more correctly, taking $\text {VAR}[\hat \mu]$ and $\text {VAR}[\hat \sigma]$ into account?
92920
I've been trying to develop an intuition based understanding of Bayes' theorem in terms of the prior, posterior, likelihood and marginal probability. For that I use the following equation: $$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$ where $A$ represents a hypothesis or belief and $B$ represents data or evidence. I've understood the concept of the posterior - it's a unifying entity that combines the prior belief and the likelihood of an event. What I don't understand is what does the likelihood signify? And why is the marginal probability in the denominator? After reviewing a couple of resources I came across this quote:     The likelihood is the weight of event $B$ given by the occurrence of $A$ ... $P(B|A)$ is the posterior probability of event $B$ , given that event $A$ has occurred.   The above 2 statements seem identical to me, just written in different ways. Can anyone please explain the difference between the two?
52818
So I want to fit a mixture model  $$f(y) = \pi_1 f_1 (y) + \pi_2 f_2 (y)$$  where $\pi_k = P(S = k)$ and $S_i$ is a latent unobserved variable.  I assume that, conditional on $S=k$, we have the model  $$y_i = \mu_k + \beta_k x_i + \epsilon_i$$  where $\epsilon_i \sim N_p (0, \Sigma_k)$.  My goal is to estimate parameters $\mu_k, \beta_k, \Sigma_k, \pi_k$ for each $k$.    Firstly, As I understand it, under this model we have likelihood function  $$L(\theta |y) = \prod_i \left( \pi_1 \phi_{1,p} \left(y_i-\mu_1 -\beta_1 x_i\right) + (1-\pi_1) \phi_{2,p} \left(y_i-\mu_2 -\beta_2 x_i\right)     \right)$$  where $\phi_{k,p} \sim N_p(0, \Sigma_k)$.    Secondly, in order to obtain point estimates of the parameters in a Bayesian framework, I need to obtain a posterior sample from the posterior distribution  $$p(\mu, \beta, \Sigma, \pi | y)$$  But as this is difficult, an alternative would be to use Gibbs sampling with posteriors  $$p(\mu |  \beta, \Sigma, \pi , y)$$  $$p(\beta |  \mu, \Sigma, \pi , y)$$  $$p(\Sigma |  \mu, \beta, \pi , y)$$  $$p(\pi |  \mu, \Sigma, \beta , y)$$  What I don't understand, is how to get from the given model equation and likelihood function to those posteriors. Once I have those posteriors, I am confident I can impliment Gibbs sampling.  Since my likelihood is not multivariate normal, it is a mixture of multivariate normals, I am unsure how to proceed, as there are no conjugates listed on the Wikipedia page for mixture distributions. Any help/advice is much appreciated!
47732
Let $X$ and $Y$ be independent $N(0; 1)$ random variables. Let $Z = \sqrt{X^2+Y^2}$.  (a) Derive the marginal pdf of $Z$ and then using the marginal pdf to compute ${\rm E}[Z^2]$  (b) Can you propose a different way other than that in (a) to compute ${\rm E}[Z^2]$  (c) Compute ${\rm E}[Z]$.  This is the whole question that I was asked. I can do (c) but I don't know how to find the marginal pdf of $Z$ with the given information and can't seem to find any formulas. Any help would be appreciated.
11920
I often saw a formula, used mostly to integrate on the parameter space like:  $$ p(x|y) = \int p(x|\theta) p(\theta|y) d\theta $$ where $\theta$ is the parameter.   I am confused and I hope to explain me how is that.  I know that $p(x) = \int p(x,\theta)d\theta = \int p(x|\theta)p(\theta)d\theta $.  If I use a similar idea which in my mind means "a marginal distribution is obtain by integrating a join distribution over one variable", but with a conditional, I would expect to have:  $$ p(x|y) = \int p(x,\theta|y)d\theta = \int p(x|\theta,y)p(\theta|y)d\theta$$ which obviously not not look the same.   Any help with that?  Later edit: The first formula was not correct, the integration is over $\theta$. Thanks
96159
Over the past few days I have been trying to understand how Markov Chain Monte Carlo (MCMC) works. In particular I have been trying to understand and implement the Metropolis-Hastings algorithm. So far I think I have an overall understanding of the algorithm but there are a couple of things that are not clear to me yet. I want to use MCMC to fit some models to data. Because of this I will describe my understanding of Metropolis-Hastings algorithm for fitting a straight line $f(x)=ax$ to some observed data $D$:  1) Make an initial guess for $a$. Set this $a$ as our current $a$ ($a_0$). Also add $a$ at the end of the Markov Chain ($C$).  2) Repeat the steps bellow a number of times.  3) Evaluate current likelihood (${\cal L_0}$) given $a_0$ and $D$.  4) Propose a new $a$ ($a_1$) by sampling from a normal distribution with $\mu=a_0$ and $\sigma=stepsize$. For now, $stepsize$ is constant.  5) Evaluate new likelihood (${\cal L_1}$) given $a_1$ and $D$.  6) If ${\cal L_1}$ is bigger than ${\cal L_0}$, accept $a_1$ as the new $a_0$, append it at the end of $C$ and go to step 2.  7) If ${\cal L_1}$ is smaller than ${\cal L_0}$ generate a number ($U$) in range [0,1] from a uniform distribution  8) If $U$ is smaller than the difference between the two likelihoods (${\cal L_1}$ - ${\cal L_0}$), accept $a_1$ as the new $a_0$, append it at the end of $C$ and go to step 2.  9) If $U$ is bigger than the difference between the two likelihoods (${\cal L_1}$ - ${\cal L_0}$), append the $a_0$ at the end of $C$, keep using the same $a_0$, go to step 2.  10) End of Repeat.  11) Remove some elements from the start of $C$ (burn-in phase).  12) Now take the average of the values in $C$. This average is the estimated $a$.  Now I have some questions regarding the above steps:   How do I construct the likelihood function for $f(x)=ax$ but also for any arbitrary function? Is this a correct implementation of Metropolis-Hastings algorithm? How the selection of the random number generation method at Step 7 can change the results? How is this algorithm going to change if I have multiple model parameters? For example, if I had the model $f(x)=ax+b$.   Notes/Credits: The main structure of the algorithm described above is based on the code from an MPIA Python Workshop.
30317
This exercise comes from Koop's Bayesian Econometrics. Given $\theta$, the parameter(s) of a model (in this case $\theta$ is a scalar), $\tilde{\theta}$, the point estimate of $\theta$, and constants $c_1 &gt; 0, c_2 &gt; 0$, the goal of the exercise is to show that $\tilde{\theta} = E[\theta \mid y]$ minimizes the expected loss of the squared error function $C(\tilde{\theta}, \theta) = (\tilde{\theta} - \theta)^2$, where, to quote the book, "the expectation is taken with respect to the posterior of $\theta$", i.e. $\theta \mid y$.    Per , I'm including my solution as a separate answer, both for feedback on that solution and for other people to post their own solutions.
1419
There is a classic problem:  Suppose that $X_1,\ldots,X_n$ form an i.i.d. sample from a uniform distribution on the interval $(0,\theta)$, where $\theta&gt;0$ is unknown. I would like to find the MLE of $\theta$.  The pdf of each observation will have the form: $$ f(x\mid\theta) =  \begin{cases} 1/\theta\quad&amp;\text{for }\, 0\leq x\leq \theta\\ 0 &amp;\text{otherwise}. \end{cases} $$ The likelihood function therefore has the form: $$ L(\theta) = \begin{cases} 1/\theta^n \quad&amp;\text{for }\; 0\leq x_i \leq \theta\;\; \text{for all }i,\\ 0 &amp;\text{otherwise}. \end{cases} $$ The general solution is usually that the MLE of theta must be a value of $\theta$ for which $\theta \geq x_i$ and which maximizes $1/\theta^n$ among all such values.   The reasoning is that since $1/\theta^n$ is a decreasing function of $\theta$, the estimate will be the smallest possible value of $\theta$ such that $\theta\geq x_i$.  Therefore, the mle of $\theta$, $\hat{\theta}$, is $\max(X_1,\ldots,X_n)$.  Here, I do not understand why we cannot just differentiate the likelihood function with respect to theta and then set it equal to $0$?  Thanks!
13861
Suppose you have:  $X|N \sim \text{MN}(N,p_1,p_2...p_J)$  $N \sim \text{Poisson}(\lambda)$  Where is the marginal distribution of $X$?
118652
Let $X_1,\ldots,X_n$ be a sample from the distribution whose probability density   function is:       $$f(x)=\begin{cases}\frac13e^{-\frac13(x-\theta)}&amp;, x\ge\theta \\ 0&amp;,x&lt;\theta\end{cases}$$      Find the maximum likelihood estimator of $\theta$.   So the PDF is equal to:  $$f(x\mid \theta) = \frac{1}{3}e^{-\frac{1}{3}(x-\theta)}1_{\{x\geq \theta\}}$$  and thereby the likelihood will be:   $$\mathcal{L} = \prod_{i=1}^n f(x\mid \theta)= \prod_{i=1}^n  \frac{1}{3}e^{-\frac{1}{3}(x-\theta)}1_{\{x\geq \theta\}}$$
119742
Let $X_1, \dots, X_n$ be i.i.d. with the PDF given by $f(x \mid \theta) = 2\theta^{-1} xe^{-\frac{x^2}{\theta}}$ for $x &gt; 0$. My task is to find the UMVUE for $\theta$, and I’m given the following hint: “$U(X)=\sum_{i=1}^nX_i^2$ is sufficient for $\theta$, and $U/\theta$ is a multiple of $\chi_{2n}^2$”. It’s my understanding that to find the UMVUE, you need to derive $E[T \mid U = u]$ where $T$ is an unbiased estimator and $U$ is a complete sufficient statistic. However, I’m not sure how to make sense of the given hint. Where does the Chi-squared come from?
84589
Suppose that we have a posterior distribution $p(\theta\mid y)$ and we wish to define a transformation on $\theta$ such that $\phi = f(\theta)$. I know that generally such transformations will not affect the MLE as it is on the data space, but will result in non-invariance on the Maximum A Posteriori (MAP) estimate as it is a function of the parameter space. I am wondering if there is an example to illustrate this? Thanks.
46319
I have been trying to derive the posterior distribution in the case of weighted Bayesian regression in the case of multivariate normal distribution for a few days and have been stuck. I am not sure if this is due to lack of my linear algebra skills or if I have done something really wrong.  In my case, the distributional assumptions are:  $$ y = N (B x, W^{-1} \nabla) $$  The parameters $B$ have a normal prior.   $$ P(B) = N (B_0, \Sigma_{B}) $$  The conditional posterior distribution for $B$ is proportional to $P(y|B) P(B)$. This is proportional to:  $$ \exp \big((B - B_0)^T \Sigma (B - B_0) + (y - Bx)^T \nabla (y - Bx)\big) $$  I have seen the textbooks mention something called completing the square which as far as I can tell equates to matching like terms. Even though I could work out for scalar variance terms, I am having trouble figuring out how to deal with the precision matrices and how this can be decoupled.   [EDIT]:  So, I tried to make a run with Glen_b's suggestion and came this far:  Original expression without the exponential (I will simplify notation slightly):  $$ (B - B_0)^T \Sigma (B - B_0) + ( y - Bx)^T \nabla (y - Bx) $$ Expanding:  $$ B^T\Sigma B - 2 B_0^T \Sigma B + B_0^T \Sigma B_0 + y^T \nabla y - 2 y^T \nabla B x + (Bx)^T \nabla Bx $$  $$ = B^T(\Sigma + x^T \nabla x)B - 2 (B_0^T \Sigma B + y^T \nabla B x) + C $$  So I am not sure how to make further progress with the second term. I guess I have to make it in the form 2(some_term)B but not sure how that can be done.
102052
I am looking for uninformative priors for beta distribution to work with a binomial process (Hit/Miss). At first I thought about using $\alpha=1, \beta=1$ that generate an uniform PDF, or Jeffrey prior $\alpha=0.5, \beta=0.5$. But I am actually looking for priors that have the minimum effect on posterior results, and then I thought about using an improper prior of $\alpha=0, \beta=0$. The problem here is that my posterior distribution only works if I have at least one hit and one miss. To overcome this I then thought about using a very small constant, like $\alpha=0.0001, \beta=0.0001$, just to assure that posterior $\alpha$ and $\beta$ will be $&gt;0$.  Does anyone knows if this approach is acceptable? I see numerical effects of changing these prior, but someone could give me a sort of interpretation of putting small constants like this as priors?
31178
$X_1, X_2, . . . , X_n$ are i.i.d. observations from a uniform distribution on the interval $[\theta − \frac{1}{2}, \theta + \frac{1}{2}]$.  Show that any $\theta$ between $X_{max} − \frac{1}{2}$ and $X_{min} + \frac{1}{2}$  maximizes the likelihood, and therefore, can be taken as the MLE.  I am confused...because $f(x|\theta) = \frac{1}{(\theta+\frac{1}{2}) -(\theta -\frac{1}{2})} = 1$ for $\theta-\frac{1}{2} \le x \le\theta+\frac{1}{2}$ and 0 otherwise... Isn't the likelihood function $L(x|\theta) = 1$?  The how can it be maximized between $X_{max} − \frac{1}{2}$ and $X_{min} + \frac{1}{2}$ ?