I don't get why it is so hard to get my neural network to learn such a simple function. I've tried all sorts of combinations of layer numbers, number of neurons but it doesn't seem to want to learn. Any suggestions would be greatly appreciated! Here is my code: import numpy as np import numpy from tensorflow import keras from tensorflow.keras import layers import random  inputs = keras.Input(shape=(1,)) dense = layers.Dense(50, activation=&quot;relu&quot;) x = dense(inputs) x = layers.Dense(50, activation=&quot;relu&quot;)(x) x = layers.Dense(50, activation=&quot;relu&quot;)(x) x = layers.Dense(50, activation=&quot;relu&quot;)(x) x = layers.Dense(50, activation=&quot;relu&quot;)(x) outputs = layers.Dense(1)(x)  model = keras.Model(inputs=inputs, outputs=outputs, name=&quot;kinetic_energy&quot;) adam = keras.optimizers.Adam(lr=0.0002,beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False,) model.compile(     loss='mean_squared_error',     optimizer='adam',     metrics=[&quot;mean_squared_error&quot;], )    for j in range(10):      x = np.array([])     y = np.array([])     for i in range(1000):         randint = random.randint(0, 10000)         square = randint * randint         x = numpy.append(x, randint)         y = numpy.append(y, square)     model.fit(x, y, epochs = 100)           x = np.array([20]) z = model.predict(x) print (z) ```
124271
In spirit of the famous  joke and  I started to think, if it's possible to design a neural network that implements $y = x^2$ function?  Given some representation of a number (e.g. as a vector in binary form, so that number 5 is represented as [1,0,1,0,0,0,0,...]), the neural network should learn to return its square - 25 in this case.  If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=\sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.  Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.  Can you suggest any ideas or reading on subject?
63480
I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of hyperparameters and I'm very confused on how to choose them. Also using cross-validation is not an option since training really takes a lot of time!
84976
I understand that the architecture of Convolutional Neural Networks (CNN) and Feed forward (FNN) are quite different. And that CNNs use pooling and filters of shared weights over a patch of the image. I am not so clear on the core convolution operator (1):   If anyone could link me to an explanation, I have looked at the Colah blog and Nielsen's online book, and I understand what it is doing but don't understand the convolution operator.  Also, it looks quite similar to the the core FNN function is there and difference? (2).  (1) convolution operator is $a^1 = \sigma(b + w*a^0)$ which is equivalent to:  $a^1 = \sigma(b + \sum^4_{l=0}\sum^4_{l=0}w_{l,m}a_{j+l,k+m})$  (2) feed forward operation:  $a^1_i = \sigma(\sum^n_{j=1}w_{ij}x_j +b_i)$  Many thanks  Sources: functions taken from:
63161
Whats a quick way to work the below (and problems similar) out?   For me to take this derivative it involves a lot of time and boring calculation, there has to be a better way.    This is taken from the solutions to Chris Bishop's Pattern Recognition and Machine learning:
63510
Why are bias nodes used in neural networks?  How many you should use?  In which layers you should use them: all hidden layers and the output layer?
120882
I want to learn Neural Networks.  I am a Computational Linguist. I know statistical machine learning approaches and can code in Python.   I am looking to start with its concepts, and know one or two popular models which may be useful from a Computational Linguistics perspective.  I browsed the web for reference and found a few books and materials.   Ripley, Brian D. (1996) Pattern Recognition and Neural Networks, Cambridge Bishop, C.M. (1995) Neural Networks for Pattern Recognition, Oxford: Oxford University Press. some links, like ,  (University of Toronto Psychology Department),  (University of Wisconsin Computer Science) and  (Facebook Research).   Coursera courses are generally nice, if anyone knows anything relevant from them. I prefer materials with lucid language and ample examples.
28001
I have found many answers to the question of what an epoch is, but none to this:  Why do we use multiple epochs when training a neural net? How does this not lead to overfitting?  From my understanding 2 epochs means we training with each data point using SGD at least twice.   Why do we not do this when training regression models?
23370
I have a small sub-question to .   I understand that when back-propagating through a max pooling layer the gradient is routed back in a way that the neuron in the previous layer which was selected as max gets all the gradient. What I'm not a 100% sure of is how the gradient in the next layer gets routed back to the pooling layer.  So the first question is if I have a pooling layer connected to a fully connected layer - like the image below.    When computing the gradient for the cyan "neuron" of the pooling layer do I sum all the gradients from the FC layer neurons? If this is correct then every "neuron" of the pooling layer has the same gradient?  For example if the first neuron of FC layer has a gradient of 2, second has a gradient of 3, and third a gradient of 6. What are the gradients of the blue and purple "neurons" in the pooling layer and why?  And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then? See the example below.    For the topmost rightmost "neuron" of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?  How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?  Please do not post a bunch of equations and tell me that my answer is right in there because I've been trying to wrap my head around equations and I still don't understand it perfectly that's why I'm asking this question in a simple way.
20350
LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from ) correspond to the loop around cell.  (source: ) And I understand that that part can be seen as a sort of identity function, so the derivative is one and the gradient stays constant. What I don't understand is how it does not vanish due to the other activation functions ? The input, output and forget gates use a sigmoid, which derivative is at most 0.25, and g and h were traditionally tanh. How does backpropagating through those not make the gradient vanish ?
43277
Why are activation functions of  (ReLU) considered non-linear?   $$ f(x) = \max(0,x)$$  They are linear when the input is positive and from my understanding to unlock the representative power of deep networks non-linear activations are a must, otherwise the whole network could be represented by a single layer.
25037
What is the precise definition of layer in neural network? Are things like concatenate functions, activations, batch normalizations, skip connections considered as layers?
75253
To illustrate my question, suppose that I have a training set where the input has a degree of noise but the output does not, for example;  # Training data [1.02, 1.95, 2.01, 3.06] : [1.0] [2.03, 4.11, 5.92, 8.00] : [2.0] [10.01, 11.02, 11.96, 12.04] : [1.0] [2.99, 6.06, 9.01, 12.10] : [3.0]   here the output is the gradient of the input array if it were noiseless (not the actual gradient).  After training the network, the output should look something like this for a given input.  # Expected Output [1.01, 1.96, 2.00, 3.06] : 95% confidence interval of [0.97, 1.03] [2.03, 4.11, 3.89, 3.51] : 95% confidence interval of [2.30, 4.12]   My question is how can a neural network be created such that it will return a predicted value and a measure of confidence, such as a variance or confidence interval?
90523
I was following the backpropagation tutorial by Michael Nielsen on , one of the very few places where the backprop algorithm is nicely explained both in theory and practice with an actual implementation in python.  Now, something is not clear when he explains how to calculate the error on a specific layer in terms of error in the next layer (I will provide a screenshot because I don't want to mess with notation):    Here, delta^l is the error in layer l, the circled dot is the elementwise product betweeen vectors, w is a weight matrix, and sigma primed indicates the derivative of an activation function.  Now, observe he points out that by applying the transpose operator to the next-layer-error vector (which is the previous layer during backprop), one is "sending back" the error across the network, and then, multiplying by the derivative of sigma, one applies the chain rule which completes the backprop step.  I have troubles understanding what does he really mean by saying that the traspose operator "sends back".  Perhaps he wants to apply the inverse trasformation? But unless the matrix happens to be orthogonal, which is not the case AFAIK, the transpose does NOT equal the inverse..  In any case, I would have thought of applying some kind of differential operator to matrix w, and not the transpose one.  Opinions?  NOTE: Please let me know whether this post is correctly suited for stackoverflow, or should it be posted on stats or math stackexchange.
102919
I am training a neural network using i) SGD and ii) Adam Optimizer. When using normal SGD, I get a smooth training loss vs. iteration curve as seen below (the red one). However, when I used the Adam Optimizer, the training loss curve has some spikes. What's the explanation of these spikes?   Model Details:  14 input nodes -> 2 hidden layers (100 -> 40 units) -> 4 output units  I am using default parameters for Adam beta_1 = 0.9, beta_2 = 0.999, epsilon = 1e-8 and a batch_size = 32.  i) With SGD  ii) With Adam
49384
I'm using Python Keras package for neural network. This is the . Is batch_size equals to number of test samples? From Wikipedia we have  information:     However, in other cases, evaluating the sum-gradient may require   expensive evaluations of the gradients from all summand functions.   When the training set is enormous and no simple formulas exist,   evaluating the sums of gradients becomes very expensive, because   evaluating the gradient requires evaluating all the summand functions'   gradients. To economize on the computational cost at every iteration,   stochastic gradient descent samples a subset of summand functions at   every step. This is very effective in the case of large-scale machine   learning problems.   Above information is describing test data? Is this same as batch_size in keras (Number of samples per gradient update)?
114851
Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers?   Shouldn't the algorithm calculate the error and work from there, despite the fact that the weights are initially the same?
43741
When used as an activation function in deep neural networks The ReLU function outperforms other non-linear functions like tanh or sigmoid . In my understanding the whole purpose of an activation function is to let the weighted inputs to a neuron interact non-linearly. For example, when using $sin(z)$ as the activation, the output of a two input neuron would be:  $$ sin(w_0+w_1*x_1+w_2*x_2) $$   which would approximate the function  $$  (w_0+w_1*x_1+w_2*x_2) - {(w_0+w_1*x_1+w_2*x_2)^3 \over 6} + {(w_0+w_1*x_1+w_2*x_2)^5 \over 120} $$  and contain all kinds of combinations of different powers of the features  $x_1$ and $x_2$.   Although the ReLU is also technically a non-linear function, I don't see how it can produce non-linear terms like the $sin(), tanh()$ and other activations do.  Edit: Although my question is similar to , I'd like to know how even a cascade of ReLUs are able to approximate such non-linear terms.