I was reading this article in wikipedia related to MAP . However, I had this confusion when it says  MAP estimation is a limit of Bayes estimators (under the 0-1 loss function), it is not very representative of Bayesian methods in general.  What does it mean MAP estimation is a limit and it has presented a criticism of MAP stating that the mean/median estimates will be better. I didn't get why it is so? Any guidance will be appreciated.
15808
My notes are rather light when it comes to this topic. I understand that the bayesian estimator, defined as (for sample space $\hat{x}$):  $E[\Theta | \hat{x}] = \int_{ \forall \Theta}yf_{\theta|\hat{x}}(y|\hat{x})dy $ (ie. the mean of the posterior distribution).  You can then look at loss functions where the form of the loss function determines that of $\theta^{'}$ (the estimator of $\theta$). By setting the loss function as a quadratic, absolute error or zero-one you get $\theta^{'}$ as the mean of the posterior, the median of the posterior and the mode of the posterior respectively.   The first two proofs make sense to me but it is the 3rd that I am not sure about. This is my understanding:  $E[l] = \int l(\theta^{'}, \theta)f(\theta|\hat{x})d\theta = \int (\theta^{'} - \theta)f(\theta|\hat{x})d\theta $  If $\theta^{'} = \theta \Rightarrow E[l] = 0$  If $\theta^{'} \neq \theta \Rightarrow \int (\theta^{'} - \theta)f(\theta|\hat{x})d\theta = 1 $ (as integrating over whole domain of $\theta$).  This proof obviously isn't rigorous, (and it could be incorrect too). What I can't see regardless is how this can be regarded as the 'mode' of the posterior distribution.   Thanks in advance!
95717
When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function?  Also, is there a better way to articulate this reasoning?
22576
Suppose that a signal or observation $s_1$ is drawn from the normal distribution $\mathcal{N}(\mu,\sigma^2)$, where $\sigma^2$ is known but $\mu$ is not. We want to estimate $\mu$ based on $s_1$.  Suppose further we have a normal prior distribution for $\mu$, which is $\mathcal{N}(\mu_0,\sigma_0^2)$.  In this case it is easy to determine the posterior distribution given $s_1$, which is normal (normal is a conjugate prior), and the posterior mean is also easy to determine as $$E[\mu|s_1]=\frac{\mu_0/\sigma_0^2+s_1/\sigma^2}{1/\sigma_0^2+1/\sigma^2}.$$  Now to my question: But what if we cannot observe $s_1$ directly; instead, we only know whether the realization of $s_1$ is above or below a certain threshold $t\in\mathbb{R}$. That is, instead of observing $s_1$, we only observe $\mathbf{1}\{s_1\ge t\}$ ($\mathbf{1}$ is the indicator function).  Since the "evidence" is now an interval rather than a point realization, how to compute the posterior mean $E[\mu|\mathbf{1}\{s_1\ge t\}]$? Is the posterior distribution even normal? I am at a loss here. Any help or references to help would be greatly appreciated.  Edit: I computed the posterior distribution numerically. See the plot below (where the "signal" indicates the realization is above a threshold). The posterior density is clearly not symmetric, hence not normal. So the question remains: Is there a closed form expression for the posterior density, or a somewhat simple expression for the posterior mean?
1439
The denominator of the (unbiased) variance estimator is $n-1$ as there are $n$ observations and only one parameter is being estimated.  $$ \mathbb{V}\left(X\right)=\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)^{2}}{n-1} $$  By the same token I wonder why shouldn't the denominator of covariance be $n-2$ when two parameters are being estimated?  $$ \mathbb{Cov}\left(X, Y\right)=\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)\left(Y_{i}-\overline{Y}\right)}{n-1} $$
86995
In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:  1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?  2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?
65990
I am wondering about the different ways that Bayesian and Frequentist statistic connect with each other. I recalled that the Maximum Likelihood estimate of a parameter $\theta$ is  of that parameter. That made me wonder: Is the Mean Posterior estimate of $\theta$ an unbiased estimator? That is,  Does $\phi(x)=E(\theta\mid x)$, imply $E(\phi(x)\mid\theta)=\theta$?  Note that this is indeed a meaningful question, since $\phi(x)$, while it is a Bayesian estimator, is simply a function from the data to the real line and so can also be seen as a classical frequentist estimator. If this question cannot be answered in general, please assume the prior is uniform. If not, is there some other Bayesian estimator (i.e. a function from the posterior to $\mathbb R$) that is always an unbiased estimator in the frequentist sense?
63286
Background:   I'm trying to follow .  I understand the basics of MLE estimation: likelihood, score, observed and expected Fisher information and the Fisher scoring technique. And I know how to justify .    The question:   I can't understand even the first line of this method :(   What's the intuition behind the $z_i$ working variables defined as:  $$ z_i = \hat\eta_i + (y_i -\hat\mu_i)\frac{d\eta_i}{d\mu_i}$$  Why are they used instead of $y_i$ to estimate $\beta$?   And what's their relation with the response/link function which is the connection between $\eta$ and $\mu$  If anyone has a simple explanation or can direct me to a more basic-level text about this I would be grateful.
94391
Could someone please explain to me what is the difference in the two arrows$$\rightarrow$$ and $$\mapsto$$ For example in Probability wih Martingales (Willams)   Thank you.
118521
Let $X \sim Bin(n, p)$ and let   $$\hat p = \frac{X}{n} $$ be an estimator of the probability $(p)$ of success in this binomial distribution, where $X$ is the number of successes and $n$ is the number of trials.  My question is, why does $\text{MSE}(\hat p)\rightarrow 0$ as $p \rightarrow0 \, \text{ or }\, 1$?
29975
For example, I have historical loss data and I am calculating extreme quantiles (Value-at-Risk or Probable Maximum Loss). The results obtained is for estimating the loss or predicting them? Where can one draw the line? I am confused.
98091
From :           To demonstrate the concept of Naïve Bayes Classification, consider the   example displayed in the illustration above. As indicated, the objects   can be classified as either GREEN or RED. My task is to classify new   cases as they arrive, i.e., decide to which class label they belong,   based on the currently exiting objects.      Since there are twice as many GREEN objects as RED, it is reasonable   to believe that a new case (which hasn't been observed yet) is twice   as likely to have membership GREEN rather than RED. In the Bayesian   analysis, this belief is known as the prior probability. Prior   probabilities are based on previous experience, in this case the   percentage of GREEN and RED objects, and often used to predict   outcomes before they actually happen.      Thus, we can write:            Since there is a total of 60 objects, 40 of which are GREEN and 20   RED, our prior probabilities for class membership are:             Having formulated our prior probability, we are now ready to classify   a new object (WHITE circle). Since the objects are well clustered, it   is reasonable to assume that the more GREEN (or RED) objects in the   vicinity of X, the more likely that the new cases belong to that   particular color. To measure this likelihood, we draw a circle around   X which encompasses a number (to be chosen a priori) of points   irrespective of their class labels. Then we calculate the number of   points in the circle belonging to each class label. From this we   calculate the likelihood:            From the illustration above, it is clear that Likelihood of X given   GREEN is smaller than Likelihood of X given RED, since the circle   encompasses 1 GREEN object and 3 RED ones. Thus:                  Although the prior probabilities indicate that X may belong to GREEN   (given that there are twice as many GREEN compared to RED) the   likelihood indicates otherwise; that the class membership of X is RED   (given that there are more RED objects in the vicinity of X than   GREEN). In the Bayesian analysis, the final classification is produced   by combining both sources of information, i.e., the prior and the   likelihood, to form a posterior probability using the so-called Bayes'   rule (named after Rev. Thomas Bayes 1702-1761).            Finally, we classify X as RED since its class membership achieves the   largest posterior probability.   This is where the difficulty of my maths understanding comes in.     p(Cj | x1,x2,x...,xd) is the posterior probability of class membership, i.e., the probability that X belongs to Cj but why write it like this?   Calculating the likelihood?    Posterior Probability?    I never took math, but my understanding of naive bayes is fine I think just when it comes to these decomposed methods confuses me. Could some one help with visualizing these methods and how to write the math out in an understandable way?
8183
Suppose we have i.i.d. $n$ observations $(X_1,X_2,...X_n)$ from a   population with density $$f_\theta(x)=\begin{cases}\theta x^{\theta-1}&amp;\text{ if }0\leq x\leq 1\\0&amp;\text{otherwise.}\end{cases}$$ Note that $\theta&gt;0$. If $T_n$ denotes the   maximum likelihood estimator of $\theta$ given that $n$ is the sample   size, then show that $$\text{var}(T_n)\stackrel{n\to\infty}{\longrightarrow} 0$$   This is the problem I am trying to solve: I computed $$T_n=\dfrac{-n}{\sum_{i=1}^{n}\ln(X_i)}$$ But I am stuck in finding $\text{var}(T_n)$ and showing that $\text{var}(T_n)\to0$ as $n\to\infty$.
62306
What would the normalising  constant be of the following, or atleast an approximation? I would like to avoid sampling.  $$f(\theta)=\exp(-k_1e^{-k_2\theta^2}-\theta^2)\qquad\theta\in(-\infty,\infty), \quad k_1,k_2&gt;0$$  In the event that its not possible, what would be a valid polynomial lower bound on $\log(f(\theta))$, that would take into account $k_1$  For anyone, who is interested, I end up with this expresssion because I place a Gaussian prior over (a squared exponential) kernel parameters, and trying to use variational bayes (hence, a lower bound).
109602
This post relates to my original question , but this time focusing on a more fundamental misunderstanding of how MCMC actually works.  When using Bayesian MCMC for parameter inference with a single model, it is usual to ignore the "evidence" integral and write:  $$P(\theta | D) \propto P(D | \theta) P(\theta)$$  For numerical reasons, it is also common to work with logarithms:  $$ln[P(\theta | D)] \propto ln[P(D | \theta)] + ln[P(\theta)]$$  I've used this successfully on a number of occasions, typically using Python and . However, for reasons explained in my , I've recently been wondering about "normalising" my likelihood by dividing it by the number of observations (I realise this probably sounds odd, but please see my other question for why I thought it was worth a try).  My likelihood assumes i.i.d. Gaussian errors, so my log-likelihood is just the sum of the individual data likelihoods. My prior is uniform and I'm currently using an improper log-prior function, which simply returns zero if the parameter values are "within range" and '-inf' otherwise.   I initially assumed that dividing by the number of data points wouldn't make any difference, because the RHS is still proportional to the log-posterior:  $$ln[P(\theta | D)] \propto \frac{1}{n}ln[P(D | \theta)] + ln[P(\theta)]$$  (My prior only ever returns 0 or '-inf', so multiplying that by $1/n$ makes no difference).  However, when I re-run my MCMC, I get very different results. As a sense check, I tried a simple OLS linear regression using the same code.   With my original (not normalised) likelihood, I get sensible results: the MCMC recovers the "true" parameters and the 95% CI broadly agrees with the Frequentist results from 'statsmodels':    However, with the "normalised" likelihood, I get very different (and obviously wrong) results:    I think I understand why this is, at least from the point of view of a simple Metropolis algorithm. The probability of accepting the proposed jump, $x_{i+1}$, compared to the current location, $x_i$, is:  $$\alpha = \frac{P(x_{i+1})}{P(x_i)}$$  Which, re-writing in terms of my log-posterior, $L$, becomes:  $$\alpha = \frac{exp(L(x_{i+1}))}{exp(L(x_i))} = exp(L(x_{i+1}) - L(x_i))$$  On the other hand, for my "normalised" version, all the likelihoods are divided by $n$, so this becomes:  $$\alpha = [exp(L(x_{i+1}) - L(x_i))]^\frac{1}{n}$$  which is different, and might explain why my uncertainty bounds suddenly become much larger. (I understand that emcee is more sophisticated than just basic Metropolis, but I guess a similar principle applies?).  My question:  What am I missing/misunderstanding to reconcile these two results, please? On the one hand, it's OK to ignore the "normalising constant" in Bayes' equation, because anything proportional to the log-posterior is sufficient. On the other hand, multiplying the log-posterior by a constant $\frac{1}{n}$ seems to have a dramatic effect on my results?  Thank you!
33080
I am reading Chris Bishop's Pattern Recognition and Machine Learning textbook. I came across the term probabilistic inference several times. I have a couple of questions.   Is probabilistic inference only applicable in a graphical modelling context? What's the distinction between traditional statistical inference (p-values, confidence intervals, Bayes factors etc.) and probabilistic inference? Is this a term that's specific to the CS community or is it widely used in the statistics community as well?
33867
I was asked today by a friend to calculate a limit and I am having trouble with the question.  Denote $\sin_{1}:=\sin$ and for $n&gt;1$ define $\sin_{n}=\sin(\sin_{n-1})$. Calculate $\lim_{n\to\infty}\sqrt{n}\sin_{n}(x)$ for $x\in\mathbb{R}$ (the answer should be a function of $x$ ).  My thoughts:   It is sufficient to find the limit for $x\in[0,2\pi]$ , and it is easy to find the limit at $0,2\pi$ so we need to find the limit for $x\in(0,2\pi)$.  If $[a,b]\subset(0,\pi)$ or $[a,b]\subset(\pi,2\pi)$ we have it that then $$\max_{x\in[a,b]}|\sin'(x)|=\max_{x\in[a,b]}|\cos(x)|&lt;\lambda\leq1$$ hence the map $\sin(x)$ is a contracting map.  We know there is a unique fixed-point but since $0$ is such a point I deduce that for any $x\in(0,2\pi)$ s.t $x\neq\pi$ we have it that $$\lim_{n\to\infty}\sin_{n}(x)=0$$  So I have a limit of the form "$0\cdot\infty$" and I can't figure out any way on how to tackle it.  Can someone please suggest a way to find that limit ?  Note: I am unsure about the tags, please change them if you see fit.
93236
Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a ) of known (finite) minimum and maximum values $a$ and $b$ but of unknown parameters $\mu$ and $\sigma^2$. If $X$ followed a non-truncated distribution, the maximum likelihood estimators $\widehat\mu$ and $\widehat\sigma^2$ for $\mu$ and $\sigma^2$ from $S$ would be the sample mean $\widehat\mu = \frac{1}{N} \sum_i S_i$ and the sample variance $\widehat\sigma^2 = \frac{1}{N} \sum_i (S_i - \widehat\mu)^2$. However, for a truncated distribution, the sample variance defined in this way is bounded by $(b-a)^2$ so it is not always a consistent estimator: for $\sigma^2 &gt; (b-a)^2$, it cannot converge in probability to $\sigma^2$ as $N$ goes to infinity. So it seems that $\widehat\mu$ and $\widehat\sigma^2$ are not the maximum-likelihood estimators of $\mu$ and $\sigma^2$ for a truncated distribution. Of course, this is to be expected since the $\mu$ and $\sigma^2$ parameters of a truncated normal distribution aren't its mean and variance.  So, what are the maximum likelihood estimators of the $\mu$ and $\sigma$ parameters of a truncated distribution of known minimum and maximum values?