I put together some math to show what I'm talking about:   TL;DR: the earlier the timestep, the more number of forget gate terms present in our derivative that multiply together. If one of these is equal or close to 0, then the whole gradient dies. How is this not an issue, if we train the forget gate [weights] simultaneously? Even if we set a really large forget bias, as the training progresses it'll correct this and make the forget gate at a timestep closer to what is optimal. Eg. if learn that f_10 should be 0, then the whole thing dies, even though we're still making contributions albeit small.  Am I missing something here?
20350
LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from ) correspond to the loop around cell.  (source: ) And I understand that that part can be seen as a sort of identity function, so the derivative is one and the gradient stays constant. What I don't understand is how it does not vanish due to the other activation functions ? The input, output and forget gates use a sigmoid, which derivative is at most 0.25, and g and h were traditionally tanh. How does backpropagating through those not make the gradient vanish ?
11359
I found rectified linear unit (ReLU) praised at several places as a solution to the  for neural networks. That is, one uses max(0,x) as activation function. When the activation is positive, it is obvious that this is better than, say, the sigmoid activation function, since its derivation is always 1 instead of an arbitrarily small value for large x. On the other hand, the derivation is exactly 0 when x is smaller than 0. In the worst case, when a unit is never activated, the weights for this unit would also never change anymore, and the unit would be forever useless - which seems much worse than even vanishingly small gradients. How do learning algorithms deal with that problem when they use ReLU?
119644
The activation functions I have seen in practice are either sigmoid or tanh. Why isn't step function used? What is bad about using a step function in an activation function for neural networks? What are the effects of using step function? In what ways are sigmoid/tanh superior over step?
90200
I have been told by a professor that it is not possible to combine different activation functions within a neural network.  And I can't find any examples of anyone doing this.  However, I cannot find any good explanation why.   Conceptually, it seems to make sense:  suppose I have a set of data on some population.  Imagine the population is divided into two types of people, A and B, but I don't actually have that categorical variable in my data.  We could imagine a logistic model that maps my population data into the categorical variable.  Now suppose I'm training a neural net with one hidden layer.  It seems to me that it makes perfect sense for one node in that hidden layer to have a sigmoid function as input representing the transformation of the input variables to this latent categorical variable (now as a probability, of course).  Meanwhile, all the other nodes have a linear activation function.  And then output would be a linear function of all the nodes in the hidden layer.    I have no reason to assume this would improve prediction error.  This is what I want to know:  is it possible to estimate such a model using standard approaches?
121856
For the input training set ${ \{ ({ x }_{ i }{ y }_{ i })\}  }_{ i=1 }^{ n }$  if the loss function is L(y, f(x)), then we initialize the model $M_0$ by finding the $\gamma$ which minimizes: $$ F_0(x) = \sum _{ i=1 }^{ n }{ L{ (y }_{ i, } } \gamma ) $$  which means, for every 'x' we define a model which always gives a constant value $\gamma$  So, now, in the 1st iteration, how come we are able to calculate the derivate of Loss function with respect to the previous model's function, (which is a constant $\gamma$), as derivates with respect to constant are not defined.  Can anyone explain what I'm understanding wrong over here ?
86656
I am looking for a book that goes through the mathematical aspects of neural networks, from simple forward passage of multilayer perceptron in matrix form or differentiation of activation functions, to back propagation in CNN or RNN (to mention some of the topics).  Do you know any book that goes in depth into this theory? I've had a look at a couple (such as Pattern Recognition and Machine Learning by Bishop or Deep Learning by Goodfellow, Bengio and Courville) but still have not found a rigorous one (with exercises would be a plus). Do you have any suggestions?
100048
I saw a paper () where they learn L based on the following cost function: $$(D_1L(q(k-1), q(k)) + D_2L(q(k),q(k+1)))² = 0$$ Here $D$ is the derivative with respect to ith argument of L. This can be obtained via automatic differentiation and it is fully compatible with backpropagation. So the target is always zero independently of the input of L but how does the optimization method knows the real values of L for a given input? I mean as long as the cost function is satisfied with a zero value the model is learning but it can be learning wrong values for L. For instance, if the real value of $D_1L(q(k-1), q(k))$ is 10 then the real value of $D_2L(q(k),q(k+1))$ has to be -10. However the neural network may be learning wrong values for L as long as they are symmetric, for example -3 and 3. This is an ill posed problem due to the infinite possible solutions. Am I missing something or the method in that paper simply doesn't make any sense? Is it possible to learn a neural network in regression task with constant target? Edit: The goal is not to predict always that value. The goal is to learn L by predicting always zero (see the cost function). The question here is in order to learn L in such a way that it respects the cost function, there are infinite possible solutions, for instance if $D1L(q(k−1),q(k))$ = x then D2L(q(k),q(k+1))=-x and its sum gives 0 respecting the cost function. But as you can see x can assume any value as long as the cost function is respected, when in reality the goal is to learn a specific L. But the optimization doesn't know a priori value for L so I guess it should just assume a random value for x. In the paper they don't mention the ill posedness of the problem, either they solved it or not. I would like to know if what I'm thinking makes sense or if I'm just missing something.
21608
In  it says     To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good approximation. If we don't, we might end up with $\Delta C&gt;0$, which obviously would not be good! At the same time, we don't want $\eta$ to be too small, since that will make the changes $\Delta v$ tiny, and thus the gradient descent algorithm will work very slowly. In practical implementations, $\eta$ is often varied so that Equation (9) remains a good approximation, but the algorithm isn't too slow. We'll see later how this works.   But just a few paragraphs before we established that $\Delta C\approx−\eta\nabla C⋅\nabla C=−\eta\|\nabla C\|^2$ is obviously always negative (for positive $\eta$). So how can $\Delta C$ be positive if we don't choose a small enough learning rate? What is meant there?
78224
I am trying to understand biases in neural nets, but different websites show very different answers.   For example, how many biases is there in a fully connected neural network with a single input layer with 5 units and a single output layer with 4 units? And what about a fully connected neural network with a single input layer with 5 units, a single hidden layer with 4 units, and a single output layer with 3 units?  For example, if I understand this correctly, , the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias.  However, at , it is explained such that each layer including the input-layer has one bias. So the answer to the example above is one in the first and two in the second.   What is correct? What am I misunderstanding here?
31141
When applying dropout in artificial neural networks, one needs to compensate for the fact that at training time a portion of the neurons were deactivated. To do so, there exist two common strategies:   scaling the activation at test time inverting the dropout during the training phase   The two strategies are summarized in the  below, taken from .  Which strategy is preferable, and why?    Scaling the activation at test time:    Inverting the dropout during the training phase:
17731
I'm reading , a post from the new distill journal. I'll paraphrase the main equations leading to the part which confuses me, the post describes the intuition in more detail.   The gradient descent algorithm is given by the following iterative process $$w^{k+1} = w^k-\alpha \nabla f(w^k)$$ where $w^k$ is the value of iteration $k$, the learning rate is $\alpha$ and $\nabla f(w)$ is  the gradient of the function $f$ evaluated at $w$. The function $f$ you wish to minimize.  Gradient descent with momentum is given by adding "memory" to the descent, this is described by the pair of equations:  \begin{align} z^{k+1} &amp;= \beta z^k + \nabla f(w^k) \\ w^{k+1} &amp;= w^k - \alpha z^{k+1}  \end{align}  In the next section "First Steps: Gradient Descent" the author considers a convex quadratic function  $$f(w) = \frac12w^TAw-b^Tw, \quad w \in \mathbb{R}^n, A \in \mathbb{R}^{n,n}$$ which has gradient $$\nabla f(w) = Aw-b$$ If we assume $A$ is symmetric and invertable then $f$ has optimal solution $w^\star = A^{-1}b$.   If we were to use gradient descent then we would iterate towards this optimal solution in the following way  \begin{align} w^{k+1} &amp;= w^k - \alpha \nabla f(w) \\ &amp;= w^k - \alpha (Aw^k -b) \end{align}  Then the article goes on to say "There is a very natural space to view gradient descent where all the dimensions act independently — the eigenvectors of $A$". I think this makes sense, although my intuition is kind of fuzzy.   Every symmetric matrix $A$ has an eigenvalue decomposition where $$A = Q\text{diag}(\lambda_1,\ldots,\lambda_n)Q^T.$$  Where $\lambda_1 &gt; \ldots &gt; \lambda_n$ and $Q$ is the vector with the corresponding eigenvectors as columns (right?).   This next part is where I don't understand what is going on:      If we perform a change of basis, $x^k = Q^T(w^k - w^\star)$, the   iterations break apart, becoming:       \begin{align} x_i^{k+1} &amp;= x_i^k - \alpha \lambda_i x_i^k  \\ &amp;=(1-\alpha\lambda_i)x_i^k &amp;= (1- \alpha\lambda_i)^{k+1}x_i^0 \end{align}      Moving back to our original space $w$, we can see that      $$w^k - w^\star = Qx^k = \sum\limits_{i}^n = x_i^0(1-\alpha\lambda_i)^kq_i$$   What is going on here? Where is the motivation of taking $w^k - w^\star$ into the eigendomain? What is $x^k$? Why are we now looking at invidual elements of the vector? I tried to follow the caculations through, but $x^{k+1}$ depends on $w^{k+1}$ which depends on $z^k$, which I thought we were trying to eliminate. My question is can someone expand on these few steps with some intuition and calculations? Thanks.
90523
I was following the backpropagation tutorial by Michael Nielsen on , one of the very few places where the backprop algorithm is nicely explained both in theory and practice with an actual implementation in python.  Now, something is not clear when he explains how to calculate the error on a specific layer in terms of error in the next layer (I will provide a screenshot because I don't want to mess with notation):    Here, delta^l is the error in layer l, the circled dot is the elementwise product betweeen vectors, w is a weight matrix, and sigma primed indicates the derivative of an activation function.  Now, observe he points out that by applying the transpose operator to the next-layer-error vector (which is the previous layer during backprop), one is "sending back" the error across the network, and then, multiplying by the derivative of sigma, one applies the chain rule which completes the backprop step.  I have troubles understanding what does he really mean by saying that the traspose operator "sends back".  Perhaps he wants to apply the inverse trasformation? But unless the matrix happens to be orthogonal, which is not the case AFAIK, the transpose does NOT equal the inverse..  In any case, I would have thought of applying some kind of differential operator to matrix w, and not the transpose one.  Opinions?  NOTE: Please let me know whether this post is correctly suited for stackoverflow, or should it be posted on stats or math stackexchange.
42381
I have a deep neural network model and I need to train it on my dataset which consists of about 100,000 examples, my validation data contains about 1000 examples. Because it takes time to train each example (around 0.5s for each example) and in order to avoid overfitting, I would like to apply early stopping to prevent unnecessary computation. But I am not sure how to properly train my neural network with early stopping, several things I do not quite understand now:   What would be a good validation frequency? Should I check my model on the validation data at the end of each epoch? (My batch size is 1) Is it the case that the first few epochs might yield worse result before it starts converging to better value? In that case, should we train our network for several epochs before checking for early stopping? How to handle the case when the validation loss might go up and down? In that case, early stopping might prevent my model from learning further, right?   Thank you in advance.
16634
I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar architectures and data. How can I fix this?    As for question        to which this question is inspired, the question is intentionally left general so that other questions about how to reduce the generalization error of a neural network down to a level which has been proved to be attainable, can be closed as duplicates of this one.  See also dedicated thread on Meta:
23370
I have a small sub-question to .   I understand that when back-propagating through a max pooling layer the gradient is routed back in a way that the neuron in the previous layer which was selected as max gets all the gradient. What I'm not a 100% sure of is how the gradient in the next layer gets routed back to the pooling layer.  So the first question is if I have a pooling layer connected to a fully connected layer - like the image below.    When computing the gradient for the cyan "neuron" of the pooling layer do I sum all the gradients from the FC layer neurons? If this is correct then every "neuron" of the pooling layer has the same gradient?  For example if the first neuron of FC layer has a gradient of 2, second has a gradient of 3, and third a gradient of 6. What are the gradients of the blue and purple "neurons" in the pooling layer and why?  And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then? See the example below.    For the topmost rightmost "neuron" of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?  How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?  Please do not post a bunch of equations and tell me that my answer is right in there because I've been trying to wrap my head around equations and I still don't understand it perfectly that's why I'm asking this question in a simple way.
6953
Could someone please provide an explanation for the origin of ? (Ideally someone who I have been speaking with on the h-bar)  Any advanced maths beyond basic calculus will most probably leave me at a loss, though I do not mind a challenge! Please assume little prior knowledge, as over the past few days I have discovered that much of my understanding surrounding the process as virtual particle pairs is completely wrong.
65793
I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear.  It's reasonable that one differentiates between logical and linear type functions, depending on the desired binary/continuous output but what is the advantage of sigmoid function over the simple linear one?  ReLU is especially difficult to understand for me, for instance: what is the point to use a function that behaves like linear in case of positive inputs but is "flat" in case of negatives? What is the intuition behind this? Or is it just a simple trial-error thing, nothing more?