The KL is given by: $D_{\mathrm{KL}}(P\|Q) = \int_X p \, \log \frac{p}{q} \, d\mu.$  The PDF of a Lognormal distribution is given by: $P = \frac 1 x \cdot \frac 1 {\sigma\sqrt{2\pi\,}} \exp\left( -\frac{(\ln x-\mu)^2}{2\sigma^2} \right).$  My Question is if it is possible to obtain an analytical solution for the KL with the given PDFs. I've searched the google and found nothing on it, except    Where the solution to my question is given as:  $$ D(f_i\|f_j)= \frac1{2\sigma_j^2}\left[(\mu_i-\mu_j)^2+\sigma_i^2-\sigma_j^2\right] + \ln \frac{\sigma_j}{\sigma_i}, $$ But I think this is wrong. I calculated numerically the KL for two Lognormal distributions and it does not match with this solution.  With these parameter:  $p(x)=\text{Lognormal}\left(0,1\right),~~q(x)=\text{Lognormal}\left(5,4\right)$  I get numerically:  $D_{KL_{numerical}} = 1.256$  The expression on the other hand yields:  $D_{KL_{analytical}} = 2.776$  Maybe someone else could provide more insight here because I was unable to derive an analytical solution for the KL of two Lognormal distributions and it was asked at least   If someone has some clever tricks for how to derive it, i will try my best and, if successful, post the solution here.  EDIT:  thanks to kjetil b halvorsen who pointed out that in   was named to contain the same solution for my problem as it was stated in the post i was mentioning originally.   Still the problem that i can't validate this solution persists. Can someone maybe confirm the given solution or give me a hint how to derive it ?
121061
I am trying to determine a metric that quantifies the distance between two continuous lognormal distributions. The data is actually a mixture of two lognormal distributions (I am not sure if this can be called bimodal). The individual distributions have different means but the same standard deviations. I have 20 data sets that can be divided into 4 groups, with each group having a constant standard deviation, but the 5 data sets in each group are made up of two distributions with different means. Some examples:      I am in search of a metric that accounts for the contrast (difference in means) and the standard deviation. I have tried out some parameters like coefficient of variation (of the mixture), Ashman's D etc. I have also tried percentage of overlap. However the problem always is that, at the end when plotted against the results I end up with 4 trend-lines again, corresponding to the 4 different standard deviations. Is there some other way I can quantify this?  Just to make myself clear, I want to quantify the difference between two distributions in a single case (between the red and green PDFs in the above image).
28677
Let $X_1$ and $X_2$ be two independent random variables with distribution $U(0,1)$. Let $Y=|X_1-X_2|$. Find the distribution function and density of $Y$.   Solution:  f(x1)=1 whenever x1 between 0 and 1          =0 Else where f(x2)=1 whenever x2 between 0 and 1          =0 Else where f(x1,x2)=1 whenever x1,x2 between 0 and 1           =0 Else where G(y)= P[-y &lt;= x1-x2 &lt;= y]       =\int_{-y}^y \int_{x1_y}^{y+x1} 1 dx2 dx1       =\int_{-y}^y -2y dx1       =4y^2 when y between 0 and 1       =0 when y less than 0       =1 when y greater than 1 Thus, g(y)= \diff G(y) =8y  Is it correct?
142
Let $X_1,\dots,X_{k+1}$ be mutually independent random variables, each having a gamma distribution with parameters $\alpha_i,i=1,2,\dots,k+1$ show that $Y_i=\frac{X_i}{X_1+\cdots+X_{k+1}},i=1,\dots,k$, have a joint ditribution as $\text{Dirichlet}(\alpha_1,\alpha_2,\dots,\alpha_k;\alpha_{k+1})$   Joint pdf of $(X_1,\dots,X_{k+1})=\frac{e^{-\sum_{i=1}^{k+1}x_i}x_1^{\alpha_1-1}\dots x_{k+1}^{\alpha_{k+1}-1}}{\Gamma(\alpha_1)\Gamma(\alpha_2)\dots \Gamma(\alpha_{k+1})}$.Then to find joint pdf of $(Y_1,\dots,Y_{k+1})$ I can not find jacobian i.e.$J(\frac{x_1,\dots,x_{k+1}}{y_1,\dots,y_{k+1}})$
123155
Let $X_n$ be iid random variables with $P(X_n=(-1)^kk)=\dfrac{C}{k^2\log k}$ for all $k\geq 2$. Here $C$ is just a constant so that the sum of probabilities is $1$. Find constant $\mu$ such that $S_n/n\to \mu$ in probability where $S_n=\sum_{k=1}^nX_k$.   Here $E(X_n)=\infty$. I'm a bit lost after this. In general how does one find constants like this?
36720
I know when evaluated it gives one but I can't figure out how to prove it.  Can anyone help? I believe it requires L'Hopitals rule and taking a natural log but i cannot figure out the exact math.
13985
I have learned about the intuition behind the KL Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The source I am reading goes on to say that the intuitive understanding of 'distance' between these two distributions is helpful, but should not be taken literally because for two distributions $P$ and $Q$, the KL Divergence is not symmetric in $P$ and $Q$.   I am not sure how to understand the last statement, or is this where the intuition of 'distance' breaks down?   I would appreciate a simple, but insightful example.
3975
Suppose $X$ is a random variable with pdf $f_X(x)$. Then the random variable $Y=X^2$ has the pdf   $$f_Y(y)=\begin{cases}\frac{1}{2\sqrt{y}}\left(f_X(\sqrt{y})+f_X(-\sqrt{y})\right) &amp; y \ge 0 \\ 0 &amp; y \lt 0\end{cases}$$   I understand the calculus behind this. But I'm trying to think of a way to explain it to someone who doesn't know calculus. In particular, I'm trying to explain why the factor $\frac{1}{\sqrt{y}}$ appears out front. I'll take a stab at it:  Suppose $X$ has a Gaussian distribution. Almost all the weight of its pdf is between the values, say, $-3$ and $3.$ But that maps to 0 to 9 for $Y$. So, the heavy weight in the pdf for $X$ has been extended across a wider range of values in the transformation to $Y$. Thus, for $f_Y(y)$ to be a true pdf the extra heavy weight must be downweighted by the multiplicative factor $\frac{1}{\sqrt{y}}$  How does that sound?  If anyone can provide a better explanation of their own or link to one in a document or textbook I'd greatly appreciate it. I find this variable transformation example in several intro mathematical probability/stats books. But I never find an intuitive explanation with it :(
59756
Say we have $f(x , \theta) = \frac{1}{2}e^{-|x-\theta|}$ Lets assume for simplicity, we only have 1 sample. We find that the log-likelihood for this distribution is:  $$ l(\theta , x) = -log(2) + (\theta - x) I_{(x &lt; \theta)}+(x-\theta)I_{(x \geq \theta)}. $$  Now in general, Call the score function:  $$ S(\theta , X) = Dl = \frac{Dg_{\theta}}{g_{\theta}} $$ by noticing the log-likelihood is expressed as :  $$ l(\theta, X) = log (g_{\theta}) $$  Where g is the likelihood and assume $g$ is 3 times differentiable. Furthermore, we find :  $$ DS = \frac{D^2g_\theta}{g_\theta}- SS'. $$ Which finally allows us to obtain that :  $$ var_\theta S = -E_\theta DS $$ by noticing that $E_\theta S = 0$  Now in this case:  $$ Dl = \frac{dl}{d\theta} = I_{(x &lt; \theta)} - I_{(x \geq \theta)} $$  So indeed  $$ E_\theta S = P{(x &lt; \theta)} - P{(x \geq \theta)} = 0 $$ And then $\frac{dS}{d\theta} = 0$ so $-E_\theta \frac{dS}{d\theta} = 0$  Now computing  $$ ES^2 = E( [I_{(x &lt; \theta)} - I_{(x \geq \theta)}]^2) = E(I_{(x &lt; \theta)} + I_{(x \geq \theta)}) = 1. $$ This does not match the case above, however here do not have differentiability. Is what I did correct, or plain fiction?
52575
I'm trying to make sense of a derivation I'm following from the lecture notes of Stanford's ML course.  Specifically the notes are here:   I'm particular I'm interested in the step where they are optimizing the lower bound of the log-likelihood by finding the derivative of that function w.r.t the mean (top of page 7). I'm a bit stuck on going from line 1 to line 2 in that sequence.  To copy here (using abbreviated notation for space):  $$ \nabla \mu_s \sum_{i=1}^{m}\sum_{j=1}^{k} w_j^{(i)}\log\frac{\alpha_j \exp\psi_{j}^{(i)}\phi_j}{w_j^{(i)}}  \qquad(1) $$ $$ \nabla \mu_s \sum_{i=1}^{m}\sum_{j=1}^{k} w_j^{(i)}\psi_{j}^{(i)}\qquad(2) $$ where $\psi_{j}^{(i)} =-\frac{1}{2}(x^{(i)}-\mu_j)^T\Sigma^{-1}_{j}(x^{(i)}-\mu_j)$ and $\alpha_j=\frac{1}{(2\pi)^{n/2}|\Sigma_j|^{1/2}}$.  The quantity $\mu_s$ is the mean (of a multivariate gaussian distribution) for class $s$.    In short, I'm not sure how they get from (1) to (2).  From just changing the log of (1) I see:  $$ \nabla \mu_s \sum_{i=1}^{m}\sum_{j=1}^{k} w_j^{(i)}\log \left(\alpha_j \exp\psi_{j}^{(i)}\phi_j\right) - w_j^{(i)}\log w_j^{(i)} \qquad(1) $$  To get to their form (2) my impression is that they take pre-emptively discard terms that would contain derivatives of $w_j^{(i)}$ (w.r.t. $\mu_s$).  At least if you assume that much, you get to (2).  However, as they show earlier in that PDF,  $$ w_j^{(i)} = p(z^{(i)}=j | x^{(i)}) =\frac{p(x^{(i)}|z^{(i)}=j)p(z^{(i)}=j) }{\sum_j p(x^{(i)},z^{(i)}=j) } $$ i.e. this quantity is the posterior distribution of the class labels.  Intuitively, it seems like $\partial w_j^{(i)}/ \partial \mu_s$ should be non-zero.  After all, changing the location of the mean $\mu_s$ would change your posterior probability for a particular class given the same data.  (Right?).  Just from a mechanical point of view, $\partial w_j^{(i)}/\partial \mu_j$ would seemingly have some non-zero terms since $p(x^{(i)}|z^{(i)}=j) \sim \mathcal{N}(\mu_j, \Sigma_j)$.  If I work through the algebra, I get  $$ \frac{\partial w_j^{(i)}}{\partial \mu_j} = w_j^{(i)}(1-w_j^{(i)})\frac{\partial \psi^{(i)}_{j}}{\partial \mu_j} $$  What am I missing here?  If I take the log-likelihood (without even thinking about EM algorithm),    $$ \ell = \sum_{i=1}^{m}\log \sum_{j=1}^{k}p(x^{(i)}|z^{(i)}=j)p(z^{(i)}=j) $$ and differentiate w.r.t $\mu_s$, I correctly get  $$ \mu_s^* = \frac{\sum_{i=1}^{m}w_s^{(i)}x^{(i)}}{\sum_{i=1}^{m}w_s^{(i)}} $$  so I'm comfortable with the eventual result, just not the intermediate steps.
123839
I am trying to get the maximum likelihood estimate for the parameter $p$. The distribution is the following:   $$ f(x\mid p) = \begin{cases} \frac{p}{x^2} &amp;\text{for} \ p\leq x &lt; \infty  \ \\ 0 &amp;\text{if not} \end{cases} $$  The sample has size $n$.  The problem is, when I try to estimate it by the procedure I know, I would have to estimate the likelihood function and obtain the derivative of the log-likelihood. We'd have:   $$ L(p; x) = \frac{p^n}{\prod_{i=1}^{n} x^2_i}$$  $$ \ln L(p;x) = n \ln(p) - \sum_{i=1}^{n} \ln(x_i^2) $$   For the derivative:  $$ l'(p;x) = \frac{n}{p} = 0$$   And I am stuck because it has no solution for $p$. How do I evaluate this?  Thanks!   EDIT:  So in this case I can use the indicator variable to write:   $$ L(p;x) = \frac{p^n}{\prod_{i=1}^{n}x_i^2} I_{(x_i \geq p)}$$   for $i = 1,2, \ldots n$ in the indicator variable. So the "closest" non-null value of $x \in X$ to $p$ is $min(X_1, \ldots X_n)$. Is that the point?
72982
I am trying to calculate a mean and standard deviation from 2 percentiles for a lognormal distribution.   I was successful in performing the calculation for a normal distribution using X = mean + sd * Z and solving for mean and sd.  I think I am missing an equation when I try to do the same thing for a lognormal distribution. I looked at  and trying to use ln(X) = mean + sd * Z but I'm getting confused whether the mean and sd in this case are for the normal distribution or the lognormal.  Which equations should I be using? and will I need more than 2 percentile to solve the calculations?
76134
I got a problem of calculating $E[e^X]$, where X follows a normal distribution $N(\mu, \sigma^2)$ of mean $\mu$ and standard deviation $\sigma$.  I still got no clue how to solve it. Assume $Y=e^X$. Trying to calculate this value directly by substitution $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\, e^{\frac{-(x-\mu)^2}{2\sigma^2}}$ then find $g(y)$ of $Y$ is a nightmare (and I don't know how to calculate this integral to be honest).  Another way is to find the inverse function. Assume $Y=\phi(X)$, if $\phi$ is differentiable, monotonic, and have inverse function $X=\psi(Y)$ then $g(y)$ (PDF of random variable $Y$) is as follows: $g(y)=f[\psi(y)]|\psi'(y)|$.  I think we don't need to find PDF of $Y$ explicitly to find $E[Y]$. This seems to be a classic problem. Anyone can help?
64702
I understand the Max likelihood estimators for mu and sigma for the lognormal distribution when data are actual values.  However I need to understand how these formulas are modified when data are already grouped or binned (and actual values are not available).  Specifically, for mu, the mle estimator is the sum of the logs of each X (divided by n which is the number of points).  For sigma squared, the mle estimator is the sum of (each log X minus the mu, squared); all divided by n. (Order of operations is taking each log X minus the mu; square that; sum that over all X's; then divide by n). Now suppose data in bins b1, b2, b3, and so on where b1 to b2 is the first bin; b2 to b3 second bin and so on.  What are the modified mu and sigma squared? thank you.
80015
I don't completely understand KL divergence yet, and how the breakdowns are completely different for similar looking distributions. In  it says: $$KL(q_\lambda(z|x) || p(z|x)) = E_q[log \space q_\lambda(z|x)] - E_q[log \space p(x,z)] + log \space p(x)$$ The two distributions in the comparison look similar, both stating the probability distribution of $z$ given $x$, but in the first case it's the cross entropy ($q$ relative to $q$?), in the second case we're subtracting the cross entropy of $p(x,z)$ relative to $q$ plus $log \space p(x)$. Could someone explain why this problem should be solved like this?
42104
Let $X_1, X_2, \ldots , X_n$ be a random sample from a distribution with the following pdf  $$f(x|\theta) =  \begin{cases} 1/(\theta_2−\theta_1), &amp;\quad\text {for}\quad \theta_1 \leq x\leq \theta_2\\ 0 &amp;\quad\text { otherwise}\quad \end{cases}$$ Suppose that $\theta_1$ and $\theta_2$ are unknown.  How would I go about writing the likelihood function for this distribution on $\theta_1$ and $\theta_2$.
1465
I have 2 i.i.d Uniform random variables $\theta_{1}$ and $\theta_{2}$ on $[0, 2\pi)$. Here the problem pushes us to consider that all operations are made modulo $2\pi$. My question is how to obtain the joint PDF of $\theta_{1} - \theta_{2}$ and $\theta_{1}$ and then obtain the conditional PDF of $\theta_{1}$ given $\theta_{1} - \theta_{2}$??. For the second part It can be deduced from the first by writing it as the joint PDF divided by the PDF of $\theta_{1}$. But for the first part, A common idea that I saw is to derive the $\text{CDF}(x, y) = P(\theta_{1} - \theta_{2} \le x$, $\theta_{1} \le y$) and then perform derivations respect to $x$ and $y$. Is this a right approach? If yes, how to proceed with CDF especially with our uniform distribution restrictions?
19246
The Multivariate Gaussian pdf is given by  $$(2\pi)^{-\frac{K}{2}} \det(\Sigma)^{-\frac{1}{2}} \exp({-\frac{1}{2}}(X-\mu)' \Sigma^{-1} (X-\mu)) $$  The wikipedia for multivariate Gaussians is    However I could not find a pdf for the multivariate lognormal distribution. Does it exist? If so, what is it?