I am training a model (Recurrent Neural Network) to classify 4 types of sequences. As I run my training I see the training loss going down until the point where I correctly classify over 90% of the samples in my training batches. However a couple of epochs later I notice that the training loss increases and that my accuracy drops. This seems weird to me as I would expect that on the training set the performance should improve with time not deteriorate. I am using cross entropy loss and my learning rate is 0.0002.  Update: It turned out that the learning rate was too high. With low a low enough learning rate I dont observe this behaviour. However I still find this peculiar. Any good explanations are welcome as to why this happens
21608
In  it says     To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good approximation. If we don't, we might end up with $\Delta C&gt;0$, which obviously would not be good! At the same time, we don't want $\eta$ to be too small, since that will make the changes $\Delta v$ tiny, and thus the gradient descent algorithm will work very slowly. In practical implementations, $\eta$ is often varied so that Equation (9) remains a good approximation, but the algorithm isn't too slow. We'll see later how this works.   But just a few paragraphs before we established that $\Delta C\approx−\eta\nabla C⋅\nabla C=−\eta\|\nabla C\|^2$ is obviously always negative (for positive $\eta$). So how can $\Delta C$ be positive if we don't choose a small enough learning rate? What is meant there?
60650
I am in the machine learning business for a long time, but still, this fundamental fact gets me confused, since every paper, article and/or book describe different kind of usages for validation and test sets. I aim to set my mind at ease forever on this issue:   Let's say I have a relatively small Convolutional Neural Network and I want to train it on MNIST dataset. Naturally, I want to learn the best hyperparameters for the given CNN, like the weight decay coefficient $\lambda$, the learning rate $\alpha$, etc. Naturally, MNIST has 60K training images and 10K test images. The basic practice is allocating 10K of the training set as the validation set for MNIST.   Assume that we are doing a grid search on $\lambda$ and $\alpha$ hyperparameters and assume that we have fixed a $(\lambda^*,\alpha^*)$ pair. Then we start training the CNN using SGD, using the 50K training set and measuring the performance on the 10K validation set in the end of each epoch (a complete pass over the 50K training set). Usually we either stop training if a fixed budget of epochs has been depleted or we start to lose accuracy on the validation set. And then, only for a single time in the training process of the CNN with the $(\lambda^*,\alpha^*)$ pair, we measure the real performance on the test set. We pick the one with best performance. This usage is intuitive and does not cause the network to unfairly overfit on the test set. But the problem is, we did not ever used the 10K validation set in the actual training with SGD.  So, now, what is the correct way to use the validation set here? After we finish training with the $(\lambda^*,\alpha^*)$ pair, either by running out of allowed epochs or early exiting due to overfit on validation set, should we merge the validation set and training set, and train a few epochs more? At this point, our learning rate may already be very small due to decaying it gradually. Should we train everything from scratch by using the 60K training + validation set, setting the learning rate to its initial value?   Another alternative may be to train the CNN with every $(\lambda,\alpha)$ pair in consideration of our grid search, using only 50K training samples in the SGD and 10K validation samples for measuring the accuracy. No interaction with the test set is allowed. Let's assume that after we train with every $(\lambda,\alpha)$ pair, we pick the pair which yields the highest accuracy on the validation set. Then we train the CNN with the picked hyperparameters, from scratch, this time using the 60K training + validation samples directly in SGD, for an amount of fixed epochs. After the training ends, we use the test set once and for all to declare our final accuracy performance.  This method, in my mind, would cause the following issue: The hyperparameters we have picked would be the optimal ones for the 50K training samples! Since we have used a smaller subset of the actual training set, most probably the model would more easily overfit, seeing a lower data variation, so our hyperparameter search may tend to find a higher $\lambda$. And with this $\lambda$ we would probably see a lower accuracy on the actual test set with the full 60K training samples, since the model won't be able to compensate for the variation of the extra 10K samples, due to high $\lambda$.  So, I am not certain what the ultimately correct way to use the validation-test sets. What would be the valid - logical approach here?
65378
I am playing with  since all state of the art results in machine learning seem to be using it (for example, see ). I am familiar with all the guidelines (train longer, increase capacity of the model, use higher learning rates), but still cannot see it working. I've tried several different examples: , , MLP for MNIST, , and turning off dropout makes all my results better even though the default configurations have dropout (taken from the ). For example, I am attaching my results for one of the models trained on the IRIS dataset. The configuration without dropout has clearly the best performance.   What am I missing?  The code for the IRIS example is .
120882
I want to learn Neural Networks.  I am a Computational Linguist. I know statistical machine learning approaches and can code in Python.   I am looking to start with its concepts, and know one or two popular models which may be useful from a Computational Linguistics perspective.  I browsed the web for reference and found a few books and materials.   Ripley, Brian D. (1996) Pattern Recognition and Neural Networks, Cambridge Bishop, C.M. (1995) Neural Networks for Pattern Recognition, Oxford: Oxford University Press. some links, like ,  (University of Toronto Psychology Department),  (University of Wisconsin Computer Science) and  (Facebook Research).   Coursera courses are generally nice, if anyone knows anything relevant from them. I prefer materials with lucid language and ample examples.
81007
In the paper Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift () Before explaining the process of batch normalization the paper tries to explain the issues related with (I am not getting what the exact issue addressed here is).  excerpt from section 2, para 2:     We could consider whitening activations at every training   step or at some interval, either by modifying the   network directly or by changing the parameters of the   optimization algorithm to depend on the network activation   values (Wiesler et al., 2014; Raiko et al., 2012;   Povey et al., 2014; Desjardins &amp; Kavukcuoglu). However,   if these modifications are interspersed with the optimization   steps, then the gradient descent step may attempt   to update the parameters in a way that requires   the normalization to be updated, which reduces the effect   of the gradient step. For example, consider a layer   with the input u that adds the learned bias $b$, and normalizes   the result by subtracting the mean of the activation   computed over the training data: $\hat x= x − E[x]$ where   $x = u + b, X = {x_{1...N}}$ is the set of values of $x$ over   the training set, and $E[x] = \frac 1 N(\sum_{i=1}^nx_i)$   .       If a gradient   descent step ignores the dependence of E[x] on b, then it   will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then   $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$.      Thus, the combination of the update to b and subsequent   change in normalization led to no change in the output   of the layer nor, consequently, the loss. As the training   continues, b will grow indefinitely while the loss remains   fixed. This problem can get worse if the normalization not   only centers but also scales the activations.   here is my understanding of the literature:   We have a batch of size N (One training batch) Let there be two arbitrary hidden layer connected to each other (L1 and L2) connected by parameters $W$ and $b$ output coming out of L1 is x1  $u = x1W$ (this is where the literature above starts. dimension of u is MxN)(M is the number of units in L2) $x = u+b$ (dimension b = dimension x = dimension u = MxN) Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x= x − E[x]$) We compute the loss and backpropogate the gradient and update just this layer's $b$ in order to give it a sanity test. New $b$ = $b + \Delta b$ We run it again on the same batch with updated $b$ repeat 3 and 4 $x_{new} = u+b + \Delta b$ (dimension b, $\Delta b$ = dimension x = dimension u = MxN) Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x = x + \Delta b − E[x + \Delta b] = x - E[x]$). which is the same as what was calculated before updating b and hence updating b had to effect on the training   My question is with this part of the excerpt:  "If a gradient descent step ignores the dependence of E[x] on b, then it will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$."  Why is   " $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$." dependent upon what comes before it? What is even the point of that bit? Please also note the useage of word "Then" (made bold) implying the statement necessarily draws causality from what comes before
54978
I'm training a convolutional neural network by adding small perturbations (like rotation and shifting) to the images each time it gets a batch data.  I think a better way of doing this could be to generate a bigger data set by oversampling with these transformations before training, however I don't have much RAM to store a bigger data set, so I decided to transform them on the run. But it means that now I'm supplying a different data set in each epoch.   I think it should make sense, and the training, in fact, converges to a better performance, but it watches a zig-zaggy road, which is understandable I guess. So am I safe to assume that I'm not making a mistake? Also is there a better way for this, i.e. training a generalizable classifier under a small data set and limited computational resources?  Thanks for any help !
12674
I'm trying to understand LSTM, using for instance   I get the overall idea, I guess. But I'm not quite sure I get the maths.  I'll set a very simple problem : I have a sequence of numbers and want to predict next number.   So x_t is of shape (1), and as I understand, h_t will be the prediction, so it should also be of shape (1). (I'm just here ignoring batch size)  Now, the equation producing h_t, using * operation should then have two operands of the same size as the result; that is, C_t and o_t should also be of shape (1).  Following on that idea, equation producing C_t forces also shape of (1) for f_t, i_t and ~C_t.  So... everything reduced to scalar real numbers in that case ?  what am I getting wrong ? because this would just not be able to learn much, would it ?
5573
First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only for predictions with only one class? If I were to use a categorical cross-entropy loss, which is typically found in most libraries (like TensorFlow), would there be a significant difference?  In fact, what are the exact differences between a categorical and binary cross-entropy? I have never seen an implementation of binary cross-entropy in TensorFlow, so I thought perhaps the categorical one works just as fine.
115147
As a follow-up to  I simplified even more and tried to train a single ReLU (with random weight) to a single ReLU.  This is the simplest network there is, and yet half the time it fails to converge.  If the initial guess is in the same orientation as the target, it learns quickly and converges to the correct weight of 1:      If the initial guess is "backwards", it gets stuck at a weight of zero and never goes through it to the region of lower loss:        I don't understand why.  Shouldn't gradient descent easily follow the loss curve to the global minima?  Example code:  from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, ReLU from tensorflow import keras import numpy as np import matplotlib.pyplot as plt  batch = 1000   def tests():     while True:         test = np.random.randn(batch)          # Generate ReLU test case         X = test         Y = test.copy()         Y[Y &lt; 0] = 0          yield X, Y   model = Sequential([Dense(1, input_dim=1, activation=None, use_bias=False)]) model.add(ReLU()) model.set_weights([[[-10]]])  model.compile(loss='mean_squared_error', optimizer='sgd')   class LossHistory(keras.callbacks.Callback):     def on_train_begin(self, logs={}):         self.losses = []         self.weights = []         self.n = 0         self.n += 1      def on_epoch_end(self, batch, logs={}):         self.losses.append(logs.get('loss'))         w = model.get_weights()         self.weights.append([x.flatten()[0] for x in w])         self.n += 1   history = LossHistory()  model.fit_generator(tests(), steps_per_epoch=100, epochs=20,                     callbacks=[history])  fig, (ax1, ax2) = plt.subplots(2, 1, True, num='Learning')  ax1.set_title('ReLU learning ReLU') ax1.semilogy(history.losses) ax1.set_ylabel('Loss') ax1.grid(True, which="both") ax1.margins(0, 0.05)  ax2.plot(history.weights) ax2.set_ylabel('Weight') ax2.set_xlabel('Epoch') ax2.grid(True, which="both") ax2.margins(0, 0.05)  plt.tight_layout() plt.show()     Similar things happen if I add bias: 2D loss function is smooth and simple, but if the relu starts upside down, it circles around and gets stuck (red starting points), and doesn't follow the gradient down to the minimum (like it does for blue starting points):    Similar things happen if I add output weight and bias, too.  (It will flip left-to-right, or down-to-up, but not both.)
43147
Question: I want to be sure of something, is the use of k-fold cross-validation with time series is straightforward, or does one need to pay special attention before using it?   Background: I'm modeling a time series of 6 year (with semi-markov chain), with a data sample every 5 min. To compare several models, I'm using a 6-fold cross-validation by separating the data in 6 year, so my training sets (to calculate the parameters) have a length of 5 years, and the test sets have a length of 1 year. I'm not taking into account the time order, so my different sets are :   fold 1 : training [1 2 3 4 5], test [6] fold 2 : training [1 2 3 4 6], test [5] fold 3 : training [1 2 3 5 6], test [4] fold 4 : training [1 2 4 5 6], test [3] fold 5 : training [1 3 4 5 6], test [2] fold 6 : training [2 3 4 5 6], test [1].   I'm making the hypothesis that each year are independent from each other. How can I verify that? Is there any reference showing the applicability of k-fold cross-validation with time series.
28836
I have created my own neural net which is using batch gradient descent. In other words, it trains on batches of examples all at once.  My issue is trying to figure out when to stop the training of the batch. I'll try to make things as understandable as possible since there are so many options, but please comment if you need clarification.  Each batch allows me to compute a list of output errors that correspond to the error the computer has on each example. Each error is a vector with elements equal to the number of output neurons.  Let each error vector be notated by e1, e2, e3, ... so the list of output errors is [e1, e2, ...]. Let an error vector in general be notated by e.  Now there are 2 things we can do to a vector or a list:   Average: average all of the elements in a vector or list. This will be notated by the function a(). For example to find the average error of an example we would do a(e) Check: make sure that each element in the vector or list is below a certain value. If any of the elements are too high, we continue to train on the entire batch. (This function can also be applied to a single value.) This will be notated by the function c(). For example, if we wanted to make sure the average error in e was low enough, we would do c(a(e)) Sum: add up all of the elements in the vector or list. This will be notated by the function s(). For example, s(e) would be totaling the error found in error vector e.   Here are the methods I have determined for deciding when to stop gradient descent:   Iterative: just run the training a certain number of times Average of averages: Find the average error of each example, then find the average of those and make sure it's below a certain amount. This is given by the formula c(a([a(e1), a(e2), ...])) Specific error: Check to make sure the error in each output neuron is below a certain amount. This is given by the formula c(e1), c(e2), ... Specific averages: Check to make sure the average error in each example is below a certain amount. This is given by the formula c([a(e1), a(e2), ...]) Sum of sums: Total up all the error in the entire batch and then make sure it's low enough. This is given by the formula c(s([s(e1), s(e2), ...]) Sum of averages: Find the average error of each example, then add up all those and make sure it's below a certain amount. This is given by the formula c(s([a(e1), a(e2), ...])) Average of sums: Compute the average total error in each example, and then make sure it's below a certain amount. This is given by the formula c(a([s(e1), s(e2), ...])) Specific sums: Check to make sure the total of the error in each example is below a certain amount. This is given by the formula c([s(e1), s(e2), ...])   So my question is this:  Which one of these options is most commonly used when training neural nets? Why? (Is there another option that I've missed?)  (Edit: I have found a very general version of this question , where the answers seem to say there is no specifically good way. If that is still true, please say so in your answer. Note that this is not a duplicate, as I explain the specific methods I would like to use.)  Edit: as A.D pointed out, summing is pretty useless because it depends on how many values are in each list/array/batch. And if you know how many examples there are, you can just use averaging. So, the methods involving "summing" have been cut.
111760
I have a dataset with two classes of elements. I also have two methods which assign (complementary) probabilities to each element in the dataset of belonging to either class.  Given that I work with probabilities (instead of hard 0,1 classification values), I was pointed to  as a way to asses which method performs better. The two most used rules appear to be:    (Log loss, logistic loss, cross-entropy loss)  (Brier score)   with Log loss apparently being the standard approach (is it?). I also found scikit-learn's  , an implementation of the:    (AUC, ROC-AUC)   which appears to do pretty much the same thing.  My question is: is either one of these inherently "better" than the other in some form? I also could just use all three. Is this advisable?
80592
I am currently working on training a 5-layer neural network, and I got some problems with tanh layer and would like to try ReLU layer.  But I found that it becomes even worse for ReLU layer.  I am wondering if it is due to that I did not find the best parameters or simply because ReLU is only good for deep networks?  Thanks!
104906
I have a training set with about 3000 positive instances and 3000 negative instances. But my test data set is pretty much un-balanced. The positive set only has 50 instances and negative has 1500 instances.This causes the precision very low. Are there any approaches to solve this problem? I use SVM to build classifier.
79528
For a given input into the input nodes, there are multiple correct values for the output nodes. In the training set, there are times when the inputs result in a certain output, and other times when the inputs result in a completely different (but equally valid) output.  Will a neural network still be able to "figure out" a pattern? From what I know about backpropagation, it seems like the different right answers would prevent it from functioning properly. If that's so, are there any solutions?  I don't need the neural network to predict all possible correct solutions, but I do need it to output a correct solution.
2412
I am using the ADAM optimizer at the moment with a learning rate of 0.001 and a weight decay value of 0.005.  I understand that weight decay reduces the weights values over time and that the learning rate modifies to weight in the right direction. Does it makes sense to have a higher weight decay value than learning rate? Or will the weights go faster to zero, than that they learn?
110323
I'm familiar with basic gradient descent algorithms for training neural networks. I've read the paper proposing Adam: .  While I've definitely got some insights (at least), the paper seems to be too high level for me overall. For example, a cost function $J(\theta)$ is often a sum of many different functions, therefore a vast amount of calculations have to be made to optimize its value; stochastic gradient descents - as far as I'm understanding the topic - calculate the optimization only for a subset of the these functions. To me it is unclear, how Adam does this and why this results in a decreased training error for the whole of $J(\theta)$.  I think Adam updates its gradient by taking into account the previous gradient. They call it something like utilizing the momentum? What exactly is this momentum? According to the algorithm on page two in the paper, it is some kind of moving average, like some estimates of the first and second moments of the "regular" gradient?  Practically, I would suspect that Adam enables one to use larger effective step sizes for decreasing the gradient and therefore the training error in combination with the stochastic approximation. Thus, the resulting update vector should "jump" around more in spatial dimensions, rather describing some curve like normal gradient descent algorithms would do.  Can someone de-mystify how Adam works? Especially how it converges, specifically why Adam's method work and what exactly the benefit is?