I'm currently implementing a Gaussian Naive Bayes classifier. Of course if I'm doing classification by  $$ \text{argmax}_{C_i} P(C_i)P(D|C_i),  $$  then the probabilities can get very small. So I want to use log probabilities. I'm seeing three posibilities:  $$ \text{argmax}_{C_i} P(C_i)\log P(D|C_i),  $$  $$ \text{argmax}_{C_i} \log P(C_i) \log P(D|C_i),  $$  $$ \text{argmax}_{C_i} \log P(C_i) + \log P(D|C_i),  $$  Which of them are the correct way to go? From a calculation point of view the second one is the right because for the others I'm getting negative values but from a math point of view the third one is the right due to the following:  $$ P(C_i|D) = \frac{P(C_i)P(D|C_i)}{P(D)} = P(C_i)P(D|C_i) $$  $$ \log P(C_i|D) = log[P(C_i)P(D|C_i)] = \log P(C_i) + \log P(D|C_i) $$  P(D) can be dropped because it does not depend on the class. Anyway for all variants I'm getting values outside [0,1] but I think this is ok because I'm calculating probability densitiy (from Gaussian distribution) and not probability.  I have a second question. I'm also interested in getting the importance for each feature for each pair of classes. How could this be calculated based on Gaussian Naive Bayes? I need this because I want to visualize the 10 most important features for each pair of classes.
116527
I have read about the log-sum-exp trick in many places (e.g. , and ) but have never seen an example of how it is applied specifically to the Naive Bayes classifier (e.g. with discrete features and two classes)  How exactly would one avoid the problem of numerical underflow using this trick?
42640
I am currently reading an introductory machine learning book by Daumé () and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator features, the following question is proposed:     Is it a good idea to map a categorical feature with n values to log2(n) binary features?   Why wouldn't that be the case, seeing as how much resources could be spared by working with fewer features? Does this approach depend on the model that is being used?
91434
When I studied Bayesian statistics, a question about the notation of Bayes' Theorem came to my mind. Below is the density function version of Bayes' Theorem, where $y$ is data vector and $\theta$ is the parameter vector:  $$ p(\theta|y)=\frac{p(y|\theta)p(\theta)}{p(y)} $$ The numerator on the right handside can be written as: $$ p(y,\theta) $$ which is the joint probability distribution of $y$ and $\theta$, then Bayes' theorem could be written as: $$ p(\theta|y)=\frac{p(y,\theta)}{p(y)} $$ Furthermore, $$  p(\theta|y)\propto p(y,\theta) $$ Am I right on this? I think it does not look right. Because the posterior is the proportional to the joint density function. But where is the mistake?
22825
This is from an exercise about the cost of missclassifications using a Naive Bayes classifier. We know that for some example which has three binary features $x_1,x_2,x_3$ the posterior is $P(Y = 1 | X_1 = x_1, X_2 = x_2, X_3 = x_3) = p$. There are two classes, y $\in$ {0,1}.The cost for misclassication is $c_m$. For each prediction we can either trust our naive bayes classifier, or ask a human expert with cost $c_h $ &lt; $c_m$, who is 100% accurate.   The question is which of the two procedures would  minimize the expected cost?  Edit: I think part of the confusion we are having is this: We have two possible outcomes. So normally, if y = 1 is true, but we decide on state y = 0, i.e. a = 0, the cost should be $p*c_m$. If in contrast  y = 0 is true, but we decide on state y = 1, i.e. a = 1, the cost should be $(p-1)*c_m$. Would choosing the optimal action a* not imply by itself then that we are deciding based on the higher posterior?  BUT, the the answer below suggests that we already should take into account that the classifier goes with the maximal posterior before searching for optimal action a*: In this scenario, the cost would always be $c_m*min(1-p,p)$. For instance, if p = 0.45, the Naive Bayes decides on y = 0 (because of the higher posterior for y=0), so the cost is $0.45*c_m$, but even if p = 0.55, the cost would be $0.45*c_m$ since now Naive Bayes is choosing y =1.
64910
I'm trying to wrap my head around the result of Bayes Theorem applied to the classic mammogram example, with the twist of the mammogram being perfect.  That is,  Incidence of cancer: $.01$  Probability of a positive mammogram, given the patient has cancer: $1$  Probability of a positive mammogram, given the patient does not have cancer: $.01$  By Bayes:  P(cancer | mammogram+) = $\dfrac {1 \cdot .01}{(1 \cdot .01) + (.091 \cdot .99)}$  $ = .5025$  So, if a random person from the population takes the mammogram, and obtains a positive result, there is a 50% chance they have cancer? I'm failing to intuitively understand how the tiny 1% chance of a false positive in 1% of the population can trigger a 50% result. Logically, I would think a perfectly true positive mammogram with a tiny false positive rate would be much more accurate.
48239
I've built a logistic regression classifier that is very accurate on my data.  Now I want to understand better why it is working so well.  Specifically, I'd like to rank which features are making the biggest contribution (which features are most important) and, ideally, quantify how much each feature is contributing to the accuracy of the overall model (or something in this vein).  How do I do this?  My first thought was to rank them based on their coefficient, but I suspect this can't be right.  If I have two features that are equally useful, but the spread of the first is ten times as large as the second, then I'd expect the first to receive a lower coefficient than the second.  Is there a more reasonable way to evaluate feature importance?  Note that I'm not trying to understand how much a small change in the feature affects the probability of the outcome.  Rather, I'm trying to understand how valuable each feature is, in terms of making the classifier accurate.  Also, my goal is not so much to perform feature selection or construct a model with fewer features, but to try to provide some "explainability" for the learned model, so the classifier isn't just an opaque black-box.
33080
I am reading Chris Bishop's Pattern Recognition and Machine Learning textbook. I came across the term probabilistic inference several times. I have a couple of questions.   Is probabilistic inference only applicable in a graphical modelling context? What's the distinction between traditional statistical inference (p-values, confidence intervals, Bayes factors etc.) and probabilistic inference? Is this a term that's specific to the CS community or is it widely used in the statistics community as well?
25422
I want to make my classifier prioritise finding true cases (1) even if that means that a lot of the false cases (0) are also classified as true.  Specifically I wish to find the weights to my features that gives at least 97 % of my true cases as true and as a small fraction as possible of my false cases as false.   What should I be looking for in a logistic regression setting? Of course I can just adjust the intercept but I assume that there must be some better way?
94952
I'm reading some course notes on Bayesian statistics and in one of the slides entitled 'model evidence' it writes: $$p(y|m)=\int{p(y,\theta|m)d\theta}=\int{p(y|\theta,m)p(\theta | m)d\theta}$$ &quot;Because we have marginalised over $\theta$ the evidence is also known as the marginal likelihood.&quot; I have two issues here.  I don't understand how does $p(y,\theta|m)$ become $p(y|\theta,m)p(\theta | m)$. Is this derived from the multiplication rule of dependent events $P(A,B)=P(A|B)P(B)=P(B|A)P(A)$? If so - I don't see how. What is this linked to &quot;we have marginalized over theta&quot;? What does $m$ really stand for? I know what it's just supposed to stand for but it's just beyond my comprehension. How is it related to model parameters?
58571
I have a data set with highly positively correlated features that I'm classifying with LR. AFAIK correlated weights are not a problem in the same way they are in Naive Bayes - overcounting will not occur with LR.   The strange things that I'm seeing is that some of the highly correlated features assume opposite weights: feature A might be highly positive and feature B highly negative, though not as much. Is this a symptom of something going wrong with optimization or is this expected (a priori I expect A and B to be positive class indicators)
77854
I'm studying 'Introduction to Statistical Learning' by James, Witten, Hastie, Tibshirani.   In page 139, of their book, they began by introducing Bayes' Theorem $p_k(X)=P(Y=k|X=x) = \dfrac{\pi_kf_k(x)}{\sum_{l=1}^k \pi_l f_l(x)}$. $\pi$ is not mathematical constant, but denotes the prior probability. Nothing is strange in this equation.   The book claims that it wants to obtain an estimate for $f_k(x)$ that it can plug into the equation given above. To estimate $f_k(x)$, it assumes that is normal. In the one dimensional setting, $f_k(x)=\dfrac{1}{\sqrt{2\pi}\sigma}\exp(-\dfrac{1}{2\sigma^2}(x-\mu_k)^2)$, where $\mu_k$ and $\sigma^2_k$ are the mean and variance for the $k$th class. It is assumed that $\sigma^2_1 = \sigma^2_2 = \cdots = \sigma^2_K$. (I began to get confused from the last statement.)   Plugging $f_k$ into $p_x$, you have this quite messy equation (1) :   $$p_x(k)=\dfrac{\pi_k \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_k)^2)}{\sum_{l=1}^K \pi_l \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{1}{2\sigma^2}(x-\mu_l)^2)}. $$  Again, no surprises here since it is just substitution.     The Bayes' Classifier involves assigning an observation to the class for which equation (1) is the largest. Taking the log of Equation (1) and rearranging terms, it is not hard to show that this is equivalent to assigning the observation to the class for which the following is the largest:  $$\delta_k(x)=x \cdot \dfrac{\mu_k}{\sigma^2} - \dfrac{\mu_k^2}{2\sigma^2} + \log(\pi_k) $$  Question: I do not understand where this came from, and what it means. I tried doing the log of equation and it doesn't become this. Are we taking the derivative somewhere here, since this is the largest observation?
38218
Let $X \sim Gamma(3,3)$ and $Y \sim Exp(1)$.  How do I calculate $P(X&gt;Y)$?  I believe I rewrite it as $P(X-Y&gt;0)$ but I am unsure how to calculate $X-Y$ for two different distributions?
84498
In continuation to  question  $p(h|D) = \frac{p(D|h)p(h)}{p(D)}$  $p(h) = $prior  probability of hypothesis $h$  $p(D)$ = prior probability of training data $D$  $p(h|D)$ = probability of $h$ given $D$  $p(D|h)$ = probability of $D$ given $h$  I am inputting to a machine a training data $D$ in form of a table with $n$ training instances.  The machine read only $m$ training examples among $n$ training examples in order to get a hypothesis as output.  I have the following understanding: .  $p(h) = $prior  probability of hypothesis $h$ = $\frac{\left\vert{Version- space}\right\vert}{\left\vert{Hypothesis-space}\right\vert}$  $p(D)$ = prior probability of training data $D$ = $\frac{\left\vert{Observed-training-data}\right\vert}{\left\vert{Training-data}\right\vert}$ = $\frac{{m}}{{n}}$   $p(D|h)$ = probability of $D$ given $h$ = $\frac{\left\vert{Training-examples-satisfied-by-hypothesis-h}\right\vert}{\left\vert{Training data}\right\vert}$ = $\frac{\left\vert{Training-examples-satisfied-by-hypothesis-h}\right\vert}{{n}}$  I do not know how to interpret   $p(h|D)$ = probability of $h$ given $D$  in the above forms.  Is my understanding about formulas for corresponding probabilities is correct and what is formula for $p(h|D)$?
13356
Why is KL divergence non-negative?  From the perspective of information theory, I have such an intuitive understanding:  Say there are two ensembles $A$ and $B$ which are composed of the same set of elements labeled by $x$. $p(x)$ and $q(x)$ are different probability distributions over ensemble $A$ and $B$ respectively.   From the perspective of information theory, $\log_{2}(P(x))$ is the least amount of bits that required for recording an element $x$ for ensemble $A$. So that the expectation $$\sum_{x \in  ensemble}-p(x)\ln(p(x))$$ can be interpreted as at least how many bits that we need for recording an element in $A$ on average.   Since this formula puts a lower bound on the bits that we need on average, so that for a different ensemble $B$ which brings about a different probability distribution $q(x)$, the bound that it gives for each element $x$ will surely not bit that is given by $p(x)$, which means taking the expectation, $$\sum_{x\in ensemble}-p(x)\ln(q(x))$$ this average length will surely be greater than the former one, which leads to $$\sum_{x\in ensemble }p(x)\frac{\ln(p(x))}{\ln(q(x))} &gt; 0$$  I don't put $\ge$ here since $p(x)$ and $q(x)$ are different.  This is my intuitive understanding, is there a purely mathematical way of proving KL divergence is non-negative? The problem can be stated as:  Given $p(x)$ and $q(x)$ are both positive over real line, and $\int_{-\infty}^{+\infty}p(x)dx = 1$, $\int_{-\infty}^{+\infty}q(x)dx = 1$.  Prove $$\int_{-\infty}^{+\infty}p(x)\ln\frac{p(x)}{q(x)}$$ is non-negative.  How can this be proved? Or can this be proved without extra conditions?
76021
I'm trying to understand at a deeper level the ubiquity of log-likelihood (and perhaps more generally log-probability) in statistics and probability theory.  Log-probabilities show up all over the place: we usually work with the log-likelihood for analysis (e.g. for maximization), the Fisher information is defined in terms of the second derivative of the log-likelihood, entropy is an expected log-probability, Kullback-Liebler divergence involves log-probabilities, the expected diviance is an expected log-likelihood, etc.    Now I appreciate the many practical and convenient reasons.  Many common and useful pdfs are from exponential families, which leads to elegantly simplified terms when log-transformed.  Sums are easier to work with than products (esp for differentiating).  Log-probs have a great floating point advantage over straight probs.  Log-transforming a pdf often converts a non-concave function into a concave function.  But what is the theoretical reason/justification/motivation for log-probs?  As an example of my perplexity, consider the Fisher information (FI).  The usual explanation for intuiting the FI is that the second derivative of the log-likelihood tells us how "peaked" the log-likehood is:  a highly peaked log-likelihood means the MLE is well-specified and we are relatively sure of its value, while a nearly flat log-likehood (low curvature) means many different parameter values are nearly as good (in terms of the log-likelihood) as the MLE, so our MLE is more uncertain.   This is all well-and-good, but isn't it more natural to just find the curvature of the likelihood function itself (NOT log-transformed)?  At first glance the emphasis on the log-transform seems arbitrary and wrong.  Surely we are more interested in the curvature of the actual likelihood function.  What was Fisher's motivation for working with the score function and the Hessian of the log-likelihood instead?  Is the answer simply that, in the end, we have nice results from the log-likelihood asymptotically? E.g., Cramer-Rao and normality of the MLE/posterior.  Or is there a deeper reason?
34941
I've been researching the Bayesian take on classical statistics lately. After reading about the Bayes factor I've been left wondering if power analysis is a necessity in this view of statistics. My main reason for wondering this is the Bayes factor really just appears to be a likelihood ratio. Once it's like 25:1 it seems like I can call it a night.  Am I far off? Any other reading I can do to learn more? Currently reading this book: , by W.M. Bolstad (Wiley-Interscience; 2nd ed., 2007).
23261
My question is related to the thread .  I often get negative AIC values from the software I use.  I notice it most when I'm doing time series.  But here is what I don't get.  When defining the AIC like   $$AIC = 2k-2\ln(L)$$  $L$, the likelihood, is a joint probability and to my understanding must be bound between 0 and 1.  Mathematically this implies the $AIC$ must be positive.  So I don't know what my software is giving me for the value labeled $AIC$.  Any thoughts?