In information theory, specifically in information content, I am struggling to conceptualise what the unit of measurement actually is. I have read quite a few similar  and worked through the derivation in Probability Theory: The Logic of Science, E.T Jaynes but still struggling to piece together the information.   The part I am struggling with is why the information content takes the form.  $$ I_X(x) = log\biggl(\frac{1}{p_X(x)}\biggr)$$  From what I have read so far, the use of the logarithm function makes sense, but $\bigl(\frac{1}{p_X(x)}\bigr)$ is where I am stuck. In plain english, I understand this to mean the quantity of information transmitted, or level of surprise when some event occurs. But I don't follow why this would takes this form. One idea I have trying to confirm (unsuccessfully so far) is that if that if some event $E$ occurs with probability $P$, then the amount of information received when this event occurs could be considered as $1 - P$.   If this is true, while also satisfying the requirements outlined by Shannon, is information content defined as above due to the following?  $$ I_X(x) = log\bigl(1\bigr) - log\bigl(p_X(x)\bigr)$$ $$ I_X(x) = 0 - log\bigl(p_X(x)\bigr)$$  or  $$ I_X(x) = log\biggl(\frac{1}{p_X(x)}\biggr)$$  I'm not sure if this is just assumed as being obvious, or a coincidence, but I haven't yet found an explanation that formally describes information content in this way (i.e. why information is $\frac{1}{p_X(x)}$).
3560
I have used the principle of maximum entropy to justify the use of several distributions in various settings; however, I have yet to be able to formulate a statistical, as opposed to information-theoretic, interpretation of maximum entropy. In other words, what does maximizing the entropy imply about the statistical properties of the distribution?   Has anyone run across or perhaps discovered yourself a statistical interpretation of max. entropy distributions that does not appeal to information, but only to probabilistic concepts?  As an example of such an interpretation (not necessarily true): "For an interval of arbitrary length L on the domain of the RV (assuming its 1-d continuous for simplicity), the maximum probability that can be contained in this interval is minimized by the maximum entropy distribution."  So, you see there is no talk about "informativeness" or other more philosophical ideas, just probabilistic implications.
123759
I've been trying to understand $\sigma$-algebras and how it encodes information in context of filtration. While certain parts seem clear and logical, I can't say I get the whole picture.  I'll try to explain the counter-intuition I get with the classical example of : the probability space $\Omega = \{ HH, HT, TH, TT \}$ and a r.v. $X(\omega)$ equal to the number of heads.  At times $0$, $1$ and $2$ the available information is represented using $\sigma$-algebras $\mathcal{F}_0=\{\emptyset,\Omega\}$, $\mathcal{F}_1=\{\emptyset, \Omega, \{HH,HT\},\{TH,TT\}\}$ and $\mathcal{F}_2=\{\emptyset, \Omega,\{HH,HT\},\{TH,TT\},\{HH\},\{HT\},\{TH\},\{TT\}\}$.  One can notice that $X(\omega)$ is not measurable with respect to $\mathcal{F}_0$ and $\mathcal{F}_1$, because $X^{-1}((\frac{3}{2}; +\infty))=\{HH\}$. To me it is quite surprising: intuitively $X$ makes perfect sense at all times. In particular it has an expected value at time $0$, which I interpret as that the probability and value of all outcomes $\{\omega\}$ can be computed. How do you think of a non-measurable function?  Here's another way of expressing the same confusion. The most natural choice of $\sigma$-algebra in a finite discrete case is $\mathcal{F}=2^\Omega$, and it is implicitly used in all elementary probability problems. However, this choice of $\mathcal{F}$ does not reflect the fact that some information is known or unknown, conditional probability does. Does it mean that the statement "$\sigma$-algebra is known information" make sense only in conditioning? Why is it convenient then?
123179
I'm looking for some correct notation. Consider the random variable $V$ with support $\mathcal{V}$ and probability distribution $P_V$. Consider a function $u:\mathcal{V}\rightarrow \mathbb{R}$.   Let $\mathcal{V}$ be finite and suppose I want to  compute the expected value of $u(V)$. This is   $$ \sum_{v\in \mathcal{V}} u(v)\times  P_{V}(v) $$  Suppose now instead that $P_V$ is continuous. What is the correct notation to indicate the formula of the expected value of $u(V)$ without passing through the definition of density? I though about using $$ \int_{\mathcal{V}} u(v)\text{ } dP_{V}(v) $$  Is this correct?
93663
The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals  $$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\mathrm{d}x.$$  We also say that any random variable $X$ whose distribution has density $f$ has entropy $H_f.$  (This integral is well-defined even when $f$ has zeros, because $\log(f(x))f(x)$ can be taken to equal zero at such values.)  When $X$ and $Y$ are random variables for which $Y = X+\mu$ ($\mu$ is a constant), $Y$ is said to be a version of $X$ shifted by $\mu.$  Similarly, when $Y = X\sigma$ ($\sigma$ is a positive constant), $Y$ is said to be a version of $X$ scaled by $\sigma.$ Combining a scale with a shift gives $Y=X\sigma + \mu.$  These relations occur frequently.  For instance, changing the units of measurement of $X$ shifts and scales it.  How is the entropy of $Y = X\sigma + \mu$ related to that of $X?$
7165
My professor says that writing this is convenient $$\int \frac 1x \mathrm{d}x = \ln|x| + C\tag{1}$$ but wrong, since it should be written as: $$\int \frac 1x \mathrm{d}x = \begin{cases}\ln x + C &amp;x &gt; 0\quad(\star)\\[0.2em] \ln(-x) + C &amp;x &lt; 0\end{cases}$$  I was wondering why is that the case. I thought that the two were equivalent, as one can see by the definition of absolute value. In $(\star)$ the equality sign is dropped because the logarithm is not defined in $0$, but that would be the case with $(1)$ as well.
47871
Suppose we have a random variable $X \sim f(x|\theta)$. If $\theta_0$ were the true parameter, the the likelihood function should be maximized and the derivative equal to zero. This is the basic principle behind the maximum likelihood estimator.   As I understand it, Fisher information is defined as   $$I(\theta) = \Bbb E \Bigg[\left(\frac{\partial}{\partial \theta}f(X|\theta)\right)^2\Bigg ]$$  Thus, if $\theta_0$ is the true parameter, $I(\theta) = 0$. But if it $\theta_0$ is not the true parameter, then we will have a larger amount of Fisher information.   my questions    Does Fisher information measure the "error" of a given MLE? In other words, doesn't the existence of positive Fisher information imply my MLE can't be ideal?  How does this definition of "information" differ from that used by Shannon? Why do we call it information?
112261
Everyone seems to have different definitions everywhere I look.   According to my lecturer:  \$ R_{bit} = \frac{bits}{time} \$  \$ R_{baud} = \frac{data}{time} \$  According to  :  \$ R_{bit} = \frac{data}{time} \$  \$ R_{baud} = \frac{bits}{time} \$  Which is the correct one and why? Feel free to give the origins of why it is defined as such too.  Related question: .
23640
I'm reading  where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners.  As a likelihood example, they start with a binomial distribution:  $$p(x|n,\theta) = \binom{n}{x}\theta^x(1-\theta)^{n-x}$$  and then log both sides  $$\ell(\theta|x, n) = x \ln (\theta) + (n-x)\ln (1-\theta)$$  with the rationale that:     "Because the likelihood is only defined up to a multiplicative constant of proportionality (or an additive constant for the log-likelihood), we can rescale ... by dropping the binomial coefficient and writing the log-likelihood in place of the likelihood"   The math makes sense, but I can't understand what is meant by "the likelihood is only defined up to a multiplicative constant of proportionality" and how this allows dropping the binomial coefficient and going from $p(x|n,\theta)$ to $\ell(\theta|x,n)$.  Similar terminology has come up in other questions ( and ), but it still not clear what, practically, likelihood being defined or bringing information up to a multiplicative constant means. Is it possible to explain this in layman's terms?
37822
is the negative of the sum of the probabilities of each outcome multiplied by the logarithm of probabilities for each outcome. What purpose does the logarithm serve in this equation?  An intuitive or visual answer (as opposed to a deeply mathematical answer) will be given bonus points!
58873
Does this identity hold for all events?  $$ P(A|B) = 1-P(A'|B) $$  Logically speaking, if the probability of $A$ given $B$ occurred is $X$, shouldn't the probability that $A$ does not occur, $A'$, given $B$, be similarly $1-X$?  There is a related question .  This is the closest that I could get to proving (or disproving) it:   $P(A\cap B)=P(A)-P(A' \cap B)$ $P(A|B)P(B)=P(A)-P(A'|B)P(B)$ $\therefore P(A'|B)= \frac {P(A)} {P(B)}-P(A|B)$   Are there are certain formulae which can be used to prove this? Or does the identity only hold under certain situations, and if so, what kind of situations?  Thanks.
57179
Given 2 events $E, F$, I know that $P(E | F) = \frac{P(E \cap F)}{P(F)}$. However sometimes the Bayes' theorem is used instead: $P(E | F) = \frac{P(F | E) P(E)}{P(F|E)P(E)+P(F|E^{c})P(E^{c})}$. However, when do I know to use the former, simpler definition and when do I use the Bayes' theorem? When I get a problem, how can I recognize which definition to use?
65606
If we set the  to $1$, then entropy would just be $\ln \Omega$, temperature would be measured in , and average kinetic energy would be an integer times $\frac{T}{2}$. Why do we need separate units for temperature and energy?
121772
What is the value of   $$\int\frac{1}{\log x}dx$$ I have tried many times, but failed everytime.   Can anyone help me out in solving this question.
87756
This actually triggered me in my mind from . After some playing around I notice that the relation $a^{\log_b x} = x^{\log_b a}$ is true for any valid value of $a,b$ and $x$. I am very inquisitive to see how this holds ?
34581
I came across this question as an exercise, had a brief idea, but didn't know how to proceed.     Let $X \sim N(0, 1)$. What is the p.d.f of $|X|$ ?   I know the final p.d.f looks just like the right half of the original pdf, but extended vertically for a factor of 2.  Could someone please show mathematically what the p.d.f of variable |X| is going to be, and explain briefly? Thank you.
118049
Would like to ask a question, but first i would like to say Hello Everybody in a way that plays the system, since some geniouses decided that one should not be able to say hello in a question.  The  in quantum mechanics is well known and considered one of most basic properties of natural reality. The  is also well known and also considered one of the most basic processes of natural reality.  The uncertainty principle uses and is related to . Planck's constant has the dimensions of action and in a statistical mechanics approach, also relates nicely with the partitioning of the phase-space providing the basic measure for the entropy functional ( provides a nice outline of this).  Apart from that, there are  which relate the Heisenberg Uncertainty Principle in quantum mechanics directly and intuitively to the 2nd Law of Thermodynamics.  Is this relation correct? And if so can we derive one from the other?  Thank you  PS. One can also check , which although not the same, is related in an interesting way.  UPDATE:  anna's answer is accepted since by mentioning the derivation of (part of) the 2nd law from unitary dynamics, answers the question at least in one way. Please consider this as still open so you can add another answer. There are more alternatives (and one of which is my stance, ie thermodynamics -> uncertainty)
17538
In a lecture I read this statement, no explanation was given     "Unlike entropy, mutual information is a measure on random variables, not on distributions"   Can anyone explain what this means?  From the definition, both entropy and mutual information are defined in terms of the probabilities of random variables, so I have no understanding of it.  The statement is in this lecture,    p.10 in Exercise 3.2.10