I am confused at a very trivial point about the BatchNorm. For illustrations it is a widely used graphics that BatchNorm corrects the distribution of incoming values. To calculate mean, does it average through a batch for each neurons separately or does it average both in 2 dimensions (batch &amp; layer width)?  For example, if I zero some of the neurons after BatchNorm, does this spoil the output distribution or doesn't it since each neuron has its own zero-centered distribution already? (Assuming trainable weight and scale parameter is not used.)   In other words, are the mean and variance unique to a layer, or do we calculate mean &amp; variance for each single neuron in a layer?
122277
While implementing Batch Normalization for a particular layer 'L' with 'n' hidden neurons/units in a Neural Network, we first normalize the Activation values of that layer using their respective Mean and Standard Deviation, and then apply the Scaling and Offset factor as shown:     X-norm = (X - mu)/sd X' = (Y * X-norm) + B       where    mu = mean of X and it is a (n,1) vector          sd = standard deviation of X and it is also a (n,1) vector         X = Activation values of layer 'L' with dimension (n,m) if mini-batch    size = m   X-norm = normalized X with dimension (n,m)   Y = Gamma / Scaling factor   B = Beta / Offset factor   Now my question is, what are the dimensions of Gamma and Beta ? Are they (n,1) vectors or are they (n,m) matrices ? My intuition says that since they somewhat are analogous to the Mean and Standard Deviation, they should be (n,1) vectors.
86995
In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:  1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?  2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?
41987
I am studying statistics and I am wondering when it comes to standard error or a sampling if the calculation changes when there are weights added.   I have a weighted mean:  $$\mu_{w} = \dfrac{\sum_{i=1}^Nw_ix_i}{\sum_{i=1}^Nw_i}$$ and a weighted variance calculated by:  $$s^2_{w}=\dfrac{\sum_{i=1}^Nw_i}{(\sum_{i=1}^Nw_i)^2-\sum_{i=1}^Nw_i^2}\cdot \sum_{i=1}^N(x_i-\mu)^2$$  is the sampling error still calculated as  $$\text{SE}=\sqrt{\dfrac{s^2_{w}}{n}}$$
771
Let $X_1, X_2, \ldots X_n$ be random variables with mean $\mu$ and variance $\sigma^2$. We define the sample mean $\bar{X} = \sum_{i = 1}^n X_i$.  I am having trouble understanding why the variance of $\bar{X}$ is $\sigma^2 / n$.  Here is what I worked out thus far:  $$\begin{gather} \sigma_\bar{X}^2 = E((\bar{X} - \mu)^2) = E(\bar{X}^2 - 2\bar{X}\mu + \mu^2) = E(\bar{X}^2) - 2\mu E(\bar{X}) + E(\mu^2) = E(\bar{X}^2) - \mu. \end{gather}$$  However, I'm not sure what to do with the $E(\bar{X}^2)$.
124036
I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure out how the practical implementation of a neural network actually works.   I understand the input layer and how to normalize the data, I also understand the bias unit, but when it comes to the hidden layer, what the actual computation is in that layer, and how it maps to the output is just a little foggy. I've seen diagrams with question marks in the hidden layer, boolean functions like AND/OR/XOR, activation functions, and input nodes that map to all of the hidden units and input nodes that map to only a few hidden units each and so I just have a few questions on the practical aspect. Of course, a simple explanation of the entire neural network process like you would explain to a child, would be awesome.   What computations are done in the hidden layer?  How are those computations mapped to the output layer?   How does the ouput layer work? De-normalizing the data from the hidden layer?   Why are some layers in the input layer connected to the hidden layer and some are not?
88239
The mean absolute deviation is:   $$\dfrac{\sum_{i=1}^{n}|x_i-\bar x|}{n}$$   The variance is: $$\dfrac{\sum_{i=1}^{n}(x_i-\bar x)^2}{n-1}$$   So the mean deviation and the variance are measuring the same thing, yet variance requires squaring the difference. Why? Squaring always gives a non-negative value, but the absolute value is also a non-negative value. Why isn't it $|x_i-\bar x|^2$, then? Squaring just enlarges, why do we need to do this?  A similar question is , but mine is a little different. Thanks.
57420
Show that $(X_1,X_2)$ has a bivariate normal distribution with means $\mu_1, \mu_2$, variances $\sigma _1^2 $ and $\sigma _2^2$, and correlation coefficient $\rho $ if and only if every linear combination $t_1X_1+t_2X_2$ has a univariate normal distribution with mean $t_1\mu_1+t_2\mu_2$, and variances  $t_1^2\sigma _1^2+t_2^2\sigma _2^2+2t_1t_2\rho \sigma_{12}$ where $t_1, t_2$ are non zero constants  My work:  Joint mgf of $X_1 $and $X_2=e^{[t_1\mu_1+t_2\mu_2+\frac{1}{2}(t_1^2\sigma _1^2+t_2^2\sigma _2^2+2\rho \sigma_1 \sigma_1t_1t_2 )]} $  I am confused here to find the proper grouping to get the forward result.  For the backward result, I know I have to find the m.g.f. of $t_1X + t_2Y$, for any $t_1 $and $t_2$ real,and plug in the $E(t_1X + t_2Y)$ and $Var(t_1X + t_2Y)$.  I am struggling to get the correct backward result.  Second part:   Let $Y_i=X_i/\sigma_i, i=1,2$ Show that $Var(Y_1-Y_2)=2(1-\rho)$  I have found, $Var(Y_1)=\sigma_1, Var(Y_2)=\sigma_2$ since $X_1/\sigma_1=\sigma _1$  How can I prove that $Var(Y_1-Y_2)=2(1-\rho)$? Thank you so much for your help!
76158
Say I have a (infinite) sequences like 1, 3, 2, 2, 1, 3 ...    I want to estimate their mean and variance of the sequence at time $t$.    But I won't have enough storage to keep all the data seen previously.    I know I can compute the mean by only keeping the sum of the sequence by far, then divide through by time $t$.  But how can I compute/estimate the variance (using a constant amount of storage)? Is there any trick/model for this?
99395
How does the expected value of a continuous random variable relate to its arithmetic mean, median, etc. in a non-normal distribution (eg. skew-normal)?  I'm interested in any common/interesting distributions (eg. log-normal, simple bi/multimodal distributions, anything else weird and wonderful).  I'm looking mostly for qualitative answers, but any quantitative or formulaic answers are also welcome. I'd particularly like to see any visual representations that make it clearer.
54253
My input variables have different dimensions. Some variables are decimal while some are hundreds. Is it essential to center (subtract mean) or scale (divide by standard deviation) these input variables in order to make the data dimensionless when using random forest?
75253
To illustrate my question, suppose that I have a training set where the input has a degree of noise but the output does not, for example;  # Training data [1.02, 1.95, 2.01, 3.06] : [1.0] [2.03, 4.11, 5.92, 8.00] : [2.0] [10.01, 11.02, 11.96, 12.04] : [1.0] [2.99, 6.06, 9.01, 12.10] : [3.0]   here the output is the gradient of the input array if it were noiseless (not the actual gradient).  After training the network, the output should look something like this for a given input.  # Expected Output [1.01, 1.96, 2.00, 3.06] : 95% confidence interval of [0.97, 1.03] [2.03, 4.11, 3.89, 3.51] : 95% confidence interval of [2.30, 4.12]   My question is how can a neural network be created such that it will return a predicted value and a measure of confidence, such as a variance or confidence interval?
107688
Given the mean &amp; variance of a dataset of linear measurements X ... N~(14,2), I want to find the mean of X^3.    So by taking mean as being expected value, &amp; rearranging: -  $$Var(X)=E[X^2]−(E[X])^2$$  I get   $$ E[X^2] = Var(X) + (E[X])^2$$  $$E[X^2] = SD^2 + (E[X])^2$$  How can I go from this to using an exponent of 3 to solve the original problem?
74804
Let $(X, Y)$ be distributed as a multivariate normal with parameters  $$ \mu = \begin{bmatrix} \mu_X \\ \mu_Y \end{bmatrix} \qquad \Sigma = \begin{bmatrix} \sigma_X^2 &amp; \sigma_{XY} \\ \sigma_{XY} &amp; \sigma_Y^2 \end{bmatrix}. $$   I would like to calculate $E(X | y_1 &lt; Y &lt; y_2)$, where $y_1$ and $y_2$ are constants.  From , I have managed to work out that $$ E(X | Y &gt; y_1) = \mu_X + \frac{\sigma_{XY}}{\sigma_Y} \left[\frac{\phi\left(\frac{y_1 - \mu_y}{\sigma_Y}\right)}{1 - \Phi\left(\frac{y_1 - \mu_y}{\sigma_Y}\right)}\right]\\ E(X | Y &lt; y_2) = \mu_X - \frac{\sigma_{XY}}{\sigma_Y} \left[\frac{\phi\left(\frac{y_2 - \mu_y}{\sigma_Y}\right)}{\Phi\left(\frac{y_2 - \mu_y}{\sigma_Y}\right)}\right] ,$$ where $\phi(\cdot)$ and $\Phi(\cdot)$ are, respectively, the p.d.f. and the c.d.f. of the Normal distribution. I could not figure out how to calculate $E(X | y_1 &lt; Y &lt; y_2)$, though.  I have searched for an answer in similar posts from the Stack Exchange network, but I couldn't get a clue from them that would solve this issue. For the record, here are some of them:
121051
Does it make any sense to have two (or more) neurons in a neural network with the same weights? (intuitively it makes no sense, since all the neurons would behave the same way).   Please consider both the input and hidden layers. What if the weights are equal 0?
50890
In a VAE, the encoder learns to output two vectors:  $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$  which are the mean and variances for the latent vector $\mathbf{z}$, the latent vector $\mathbf{z}$ is then calculated by:  $$\mathbf{z} = \mu + \sigma \epsilon$$  where: $\epsilon = N(0, \mathbf{I}_{z \times z})$  The KL divergence loss for a VAE for a single sample is defined as (referenced from  and ):  $$\frac{1}{2} \left[ \left(\sum_{i=1}^{z}\mu_{i}^{2} + \sum_{i=1}^{z}\sigma_{i}^{2} \right) - \sum_{i=1}^{z} \left(log(\sigma_{i}^{2}) + 1 \right) \right]$$  Though, I'm not sure how they got their results, would anyone care to explain or point me to the right resources?
114943
Let $X_1, X_2, \ldots, X_{16}$ be  independent with $N(3,4)$ distributions and $\bar{X}$ denote the sample mean. Find:   $P(-8 &lt; 2 \bar{X} &lt; -4)$. A number $K$ such that $P(-K &lt; 2 \bar{X} &lt; K) = 0.99$. $P(-10 &lt; 3 X_1+X_7 &lt; -8)$. A number $K$ such that $P(3 X_2+X_5 &lt; K) = 0.3$.   If someone could give me some help or hints on where to start with (a) or any of the problems that would be really helpful and I'd really appreciate it.
20010
I have labored under the belief that the sample median is more robust measure of central tendency than the sample mean, since it ignores outliers.  I was therefore surprised to learn (in the answer to ) that for samples drawn from a normal distribution, the variance of the sample mean is less than the variance of the sample median (at least for large $n$).  I understand mathematically why this is true.  Is there a "philosophical" way of looking at this that that would help with intuition about when to use the median rather than the mean for other distributions?    Are there mathematical tools that help quickly answer the question for a particular distribution?