I am trying to train a Deep Q Network () for a simple control task. The agent starts in the middle of a 1-dimensional line, at state 0.5. On each step, the agent can move left, move right, or stay put. The agent then receives a reward proportional to the absolute distance from its starting state. If the agent reaches state 0.35 or state 0.65, then the episode ends. So, effectively the agent is encouraged to move all the way to the left, or all the way to the right.  For now, I am just trying to train a deep neural network in Keras / Tensorflow, to predict the q-values of the three actions, in any given state. At each time step, actions are chosen randomly, so I am just trying to see if the network can predict the correct q-values, I am not trying to get the agent to actually solve the task. However, my network is struggling to learn the correct function.  In this first figure below, the dots represent the training data available to the network on each weight update. These are stored in the experience replay buffer in the standard q-learning algorithm. As can be seen, states further from the centre have higher q-values, as would be expected. Also, when the state is greater than 0.5, higher q-values are assigned to the "move right" action, as would be expected, and vice versa. So, it seems that the training data is valid.    In the next figure below, the predicted q-values are shown from my network. To get these values, I sampled uniformly from the range 0.35-0.65 to get the inputs, and then plotted the q-values for each action (the three coloured lines). The gradient is computed with respect to the difference between these predictions, and the training data in the first figure. So, these lines should match up to the points in the first figure. However, clearly they do not. They are predicting roughly the right values, but the shape of the lines are not consistent with the training data.    In the third figure below, I now plot the training loss for this network. So, it appears that training has converged, even though there is still significant error between the training data and the predictions. I am using mean squared error loss.    Finally, here are some details on my network itself. I use 1 layer of 16 hidden nodes, each with Relu activations, and the final layer has 3 nodes, one for each action. Although this is a simple network, it has enough capacity to learn the simple function in the first figure, from my prior experiments. I use the AdamOptimizer for optimisation, with a learning rate of 0.01, and I have also tried with a learning rate of 0.001 which also does not give the desired results. My batch size is 32. As with the standard Deep Q Network approach, I used a separate network for training the q values to the network used to predict the q values for the other states (as used in the Bellman equation).  So, I am puzzled as to what is going on. My network should be able to fit the training data, but it is not able to. I am not sure whether this is a problem with my optimisation, my network architecture, or my q learning algorithm. My intuition is that the learning rate is too low, because the predicted q values show a shape significantly different to the training q values, and so the network needs to be bumped significantly in the right direction. However, I am already using a learning rate of 0.01, which is very high.  Another thought I had is that the training error is pretty small (3 x 10^-4), and so it may not be possible to reduce this any more. Therefore, I would need to increase the size of the predicted q values (e.g. by increasing the scale of the reward), such that the absolute error is much larger, and therefore the optimiser can compute effective gradients. Does this make sense?  Any suggestions?
472
I'm training a neural network but the training loss doesn't decrease. How can I fix this?  I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my network's performance doesn't improve on the training set.    This question is intentionally general so that other questions about how to train a neural network can be closed as a duplicate of this one, with the attitude that "if you give a man a fish you feed him for a day, but if you teach a man to fish, you can feed him for the rest of his life." See this Meta thread for a discussion:   If your neural network does not generalize well, see:
76963
I'm currently learning about gradients, and I thought khanacademy could help me acquiring some intuition. The actual computation is clear to me, however I'm having trouble understand the intuition.    is the video I'm talking about, and the part I have a part with is 6:11. He says the gradient vector shows the direction you have to travel in the x,y-plane in order to get a maximum slope in the z-direction.  This sounds like gibberish to me. How do you have to 'travel' it? Why do you get a maximum slope? I just have no clue.
35416
I have a ROC curve like the figure below:   What can be inferred given this kind of curve in general?  What are the differences of states 1, 2 and 3 in the figure?  What causes the diagram to jump straight from state 1 to state 3? Most of the ROC curves on the internet look like the figure below:
81007
In the paper Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift () Before explaining the process of batch normalization the paper tries to explain the issues related with (I am not getting what the exact issue addressed here is).  excerpt from section 2, para 2:     We could consider whitening activations at every training   step or at some interval, either by modifying the   network directly or by changing the parameters of the   optimization algorithm to depend on the network activation   values (Wiesler et al., 2014; Raiko et al., 2012;   Povey et al., 2014; Desjardins &amp; Kavukcuoglu). However,   if these modifications are interspersed with the optimization   steps, then the gradient descent step may attempt   to update the parameters in a way that requires   the normalization to be updated, which reduces the effect   of the gradient step. For example, consider a layer   with the input u that adds the learned bias $b$, and normalizes   the result by subtracting the mean of the activation   computed over the training data: $\hat x= x − E[x]$ where   $x = u + b, X = {x_{1...N}}$ is the set of values of $x$ over   the training set, and $E[x] = \frac 1 N(\sum_{i=1}^nx_i)$   .       If a gradient   descent step ignores the dependence of E[x] on b, then it   will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then   $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$.      Thus, the combination of the update to b and subsequent   change in normalization led to no change in the output   of the layer nor, consequently, the loss. As the training   continues, b will grow indefinitely while the loss remains   fixed. This problem can get worse if the normalization not   only centers but also scales the activations.   here is my understanding of the literature:   We have a batch of size N (One training batch) Let there be two arbitrary hidden layer connected to each other (L1 and L2) connected by parameters $W$ and $b$ output coming out of L1 is x1  $u = x1W$ (this is where the literature above starts. dimension of u is MxN)(M is the number of units in L2) $x = u+b$ (dimension b = dimension x = dimension u = MxN) Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x= x − E[x]$) We compute the loss and backpropogate the gradient and update just this layer's $b$ in order to give it a sanity test. New $b$ = $b + \Delta b$ We run it again on the same batch with updated $b$ repeat 3 and 4 $x_{new} = u+b + \Delta b$ (dimension b, $\Delta b$ = dimension x = dimension u = MxN) Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x = x + \Delta b − E[x + \Delta b] = x - E[x]$). which is the same as what was calculated before updating b and hence updating b had to effect on the training   My question is with this part of the excerpt:  "If a gradient descent step ignores the dependence of E[x] on b, then it will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$."  Why is   " $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$." dependent upon what comes before it? What is even the point of that bit? Please also note the useage of word "Then" (made bold) implying the statement necessarily draws causality from what comes before
120072
qqnorm(x) plot(qnorm(seq(80)/81),sort(x))   Finding that the plots produced by the commands above are slightly different from each other, I tried this:  qqnorm(qnorm(seq(80)/81))   I got a slightly less than perfect line.  I'd have tried regressing the result of qnorm(seq(80)/81) on the variable that was plotted on the $x$-axis and plotting residuals against the predictor, expecting to see some graceful curve, but for the fact that I don't know what to use as the predictor.  Possibly such a residual plot would reveal more than just the graceful S-shaped thing I'd anticipate.  So my question is this: if the thing on the $x$-axis in the plot produced by qqnorm is not what I get from qnorm(seq(80)/81) (and what I did shows that indeed it is not), then what is it?
86646
I am building some image classification model with reasonable size data (~3K) images in both training and validation set. However, I noticed the performance on validation set is not stable. For example, here are outputs over 10 epochs (acc means accuracy binary classification on balanced data) epoch [1]: training loss: 2.27 - acc: 0.50 - val_loss: 3.29 - val_acc: 0.49 epoch [2]: training loss: 1.55 - acc: 0.50 - val_loss: 0.92 - val_acc: 0.50 epoch [3]: training loss: 1.07 - acc: 0.51 - val_loss: 1.43 - val_acc: 0.53 epoch [4]: training loss: 0.87 - acc: 0.58 - val_loss: 1.85 - val_acc: 0.61 epoch [5]: training loss: 0.59 - acc: 0.72 - val_loss: 0.58 - val_acc: 0.61 epoch [6]: training loss: 0.52 - acc: 0.79 - val_loss: 2.30 - val_acc: 0.50 epoch [7]: training loss: 0.38 - acc: 0.85 - val_loss: 0.17 - val_acc: 0.86 epoch [8]: training loss: 0.32 - acc: 0.88 - val_loss: 1.52 - val_acc: 0.60 epoch [9]: training loss: 0.21 - acc: 0.91 - val_loss: 0.14 - val_acc: 0.88 epoch [10]: training loss: 0.34 - acc: 0.88 - val_loss: 2.81 - val_acc: 0.49  We can see that in training, it seems fine, but for epoch 6 and 8 validation loss was very high, and the final epoch 10, the validation loss got so high that the model become useless. What could be the reason causing this? If it is overfitting on training data, why we are not seeing steady increase on validation loss?
5573
First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only for predictions with only one class? If I were to use a categorical cross-entropy loss, which is typically found in most libraries (like TensorFlow), would there be a significant difference?  In fact, what are the exact differences between a categorical and binary cross-entropy? I have never seen an implementation of binary cross-entropy in TensorFlow, so I thought perhaps the categorical one works just as fine.
113
I'm training a neural network for a particular problem which can be predicted with 100% accuracy. However, the problem is that results tend to vary between 99% and 100% even if I train 10 different networks and take the average. I was reading around and found that it was due to random weights initialized every time the program ran. Is it okay to set the random seed to 1 and then train 10 different networks and take the average?  This will result in less variation and easy model selection.
107088
I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g.  or ? I know that binary crossentropy can be used in binray classification problems where the ground-truth labels (i.e. $y$) are either 0 or 1 and therefore when predictions (i.e. $p$) are correct, in both cases, the loss value would be zero:  $$ BCE(y,p) = -y.log(p) - (1-y).\log{(1-p)} $$  $$ BCE(0,0) = 0, BCE(1,1) = 0 $$  However, binary crossentropy does not have a value of zero when neither of its arguments are both zero or one, which is the case for an autoencoder with ground-truth labels in range $[0,1]$ (i.e. assuming the input data has been normalized in this range). I thought a regression loss function such as mean squared error or mean absolute error must be used instead, which have a value of zero when labels and predictions are the same. What am I missing here?
103593
Assume that a,b,n are all natural numbers.  I was going to set it up as:  na = q(1)*n(b) + r(1)     where a>b  and go down the chain:  nb = q2 * r(1) + (r2)  but something seems off. Someone told me that in the first part, it should be n*r(1), but I don't understand why. May I please have some direction?
20350
LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from ) correspond to the loop around cell.  (source: ) And I understand that that part can be seen as a sort of identity function, so the derivative is one and the gradient stays constant. What I don't understand is how it does not vanish due to the other activation functions ? The input, output and forget gates use a sigmoid, which derivative is at most 0.25, and g and h were traditionally tanh. How does backpropagating through those not make the gradient vanish ?
114851
Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers?   Shouldn't the algorithm calculate the error and work from there, despite the fact that the weights are initially the same?
63161
Whats a quick way to work the below (and problems similar) out?   For me to take this derivative it involves a lot of time and boring calculation, there has to be a better way.    This is taken from the solutions to Chris Bishop's Pattern Recognition and Machine learning:
69853
I'm sorry if this is a broad question but can someone explain to me how to interpret these regression diagnostic plots? I understand the Normal Q-Q show's how normal the spread of the data is, but the other three I am confused by. For the Residuals vs Leverage, what does it mean for a point to be outside of the bounds? What does it mean if the red lines are parabolic and curved like in the first two? Why do we care about residuals, and what do they tell us? I've searched the web multiple times for some guide to interpret these and I can't get any straight forward answers. Any help is appreciated, thank you.
119859
I have a model similar to below with a continuous outcome, continuous predictor and a categorical variable. the goal of this analysis is to make inferences on the group variable. The issue is that y is bounded by x in the form of y >= -x. When the model makes predictions you can see in the plot below that the predicted value is outside the possible range. See the last plot.   How can this be remedied? A custom loss function to penalize going beyond the bound and use gradient descent regression? Or is there some other way? Thank you.   x =   rep(seq(-10,10,1),each=5)  y =  rep(0,length(x) ) weights = sample( seq(1,1000,1) ,length(x)) groups = rep( letters[1:5], times =length(x)/5  )   ## the outcome y has the relationship y &gt;= -x for(i in 1:length(x)){   y[i] = sample(  seq(-x[i],-x[i]+ ifelse(x[i]&lt;0 ,                                            sample(seq(1,10,.1),1),                                           sample(seq(5,40,.1),1) ),1)                        ,1) }   dat =data.frame(x = x, y = y, weights = weights, group = groups) model = lm(dat = dat, y ~ x + group + group * x, weights = weights) dat$FIT= predict(model, dat, allow.new.levels = TRUE)  library(ggplot2) ggplot(data = dat, aes(x = x, y = y, color = group))+geom_point( aes(size = weights))+ geom_line(data = dat, aes(x = x, y = FIT, color = group))    #data for new predictions newdata = data.frame( x = seq(-100,100,1) ,                        group = rep("c", length( seq(-100,100,1) )),                       bound = -seq(-100,100,1)                       )  ## when pridicting new data the outcome y has the relationship y &gt;= -x ## but the predicted vales are below the bound. See how red line is below blue to the left of zero new_preds  = predict(model, newdata, allow.new.levels = TRUE) ggplot(data = newdata, aes(x = x, y = new_preds, color = group))+geom_line()  +   geom_line(data = newdata, aes(x = x, y = bound, color = "green"))
89964
I'm trying to find outliers using 1.5 x interquartile range but I'm getting a negative value for the lower bound. I found Q3 and Q1 which are 13.30 &amp; 3.00 respectively but for the lower bound i get -12.45 which seem odd. Is there something I'm doing wrong?   There are 43 numbers in this chart so I just get (75/100)*43 to find Q3 and (25/100)*43 to find Q1
12328
I use LSTM network in Keras. During the training, the loss fluctuates a lot, and I do not understand why that would happen.  Here is the NN I was using initially:   And here are the loss&amp;accuracy during the training:   (Note that the accuracy actually does reach 100% eventually, but it takes around 800 epochs.)  I thought that these fluctuations occur because of Dropout layers / changes in the learning rate (I used rmsprop/adam), so I made a simpler model:   I also used SGD without momentum and decay. I have tried different values for lr but still got the same result.  sgd = optimizers.SGD(lr=0.001, momentum=0.0, decay=0.0, nesterov=False)   But I still got the same problem: loss was fluctuating instead of just decreasing. I have always thought that the loss is just suppose to gradually go down but here it does not seem to behave like that.  So:   Is it normal for the loss to fluctuate like that during the training? And why it would happen? If not, why would this happen for the simple LSTM model with the lr parameter set to some really small value?   Thanks. (Please, note that I have checked similar questions here but it did not help me to resolve my issue.)  Upd.: loss for 1000+ epochs (no BatchNormalization layer, Keras' unmodifier RmsProp):   Upd. 2: For the final graph:  model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy']) history = model.fit(train_x, train_y, epochs = 1500)   Data: sequences of values of the current (from the sensors of a robot).   Target variables: the surface on which robot is operating (as a one-hot vector, 6 different categories).  Preprocessing:   changed the sampling frequency so the sequences are not too long (LSTM does not seem to learn otherwise); cut the sequences in the smaller sequences (the same length for all of the smaller sequences: 100 timesteps each); check that each of 6 classes has approximately the same number of examples in the training set.   No padding.  Shape of the training set (#sequences, #timesteps in a sequence, #features):  (98, 100, 1)    Shape of the corresponding labels (as a one-hot vector for 6 categories):  (98, 6)   Layers:    The rest of the parameters (learning rate, batch size) are the same as the defaults in Keras:  keras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)      batch_size: Integer or None. Number of samples per gradient update. If unspecified, it will default to 32.   Upd. 3: The loss for batch_size=4:    For batch_size=2 the LSTM did not seem to learn properly (loss fluctuates around the same value and does not decrease).  Upd. 4: To see if the problem is not just a bug in the code: I have made an artificial example (2 classes that are not difficult to classify: cos vs arccos). Loss and accuracy during the training for these examples: