What is the use of Hidden layers in Neural Networks? How do we know the role of each hidden layer?
124036
I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure out how the practical implementation of a neural network actually works.   I understand the input layer and how to normalize the data, I also understand the bias unit, but when it comes to the hidden layer, what the actual computation is in that layer, and how it maps to the output is just a little foggy. I've seen diagrams with question marks in the hidden layer, boolean functions like AND/OR/XOR, activation functions, and input nodes that map to all of the hidden units and input nodes that map to only a few hidden units each and so I just have a few questions on the practical aspect. Of course, a simple explanation of the entire neural network process like you would explain to a child, would be awesome.   What computations are done in the hidden layer?  How are those computations mapped to the output layer?   How does the ouput layer work? De-normalizing the data from the hidden layer?   Why are some layers in the input layer connected to the hidden layer and some are not?
16286
I haven't seen the question stated precisely in these terms, and this is why I make a new question.  What I am interested in knowing is not the definition of a neural network, but understanding the actual difference with a deep neural network.  For more context: I know what a neural network is and how backpropagation works. I know that a DNN must have multiple hidden layers. However, 10 years ago in class I learned that having several layers or one layer (not counting the input and output layers) was equivalent in terms of the functions a neural network is able to represent (see Cybenko's ), and that having more layers made it more complex to analyse without gain in performance. Obviously, that is not the case anymore.  I suppose, maybe wrongly, that the differences are in terms of training algorithm and properties rather than structure, and therefore I would really appreciate if the answer could underline the reasons that made the move to DNN possible (e.g. mathematical proof or randomly playing with networks?) and desirable (e.g. speed of convergence?)
93917
I'm trying to understand how Recurrent Neural Networks use word embedding vectors as their inputs, and I've created the illustration below to reflect my current-state understanding. I understand that word embedding is its own learning process to reduce the dimensionality of the relationship between words in a corpus, and that the number of dimensions returned is user-defined (similar to PCA). But am I correct in understanding that once the resulting vectors are created, that each word's &quot;embedding vector&quot; in each statement observation is then fed into its own  xt input of an RNN, when performing something like sentiment analysis? The application of the highlighted yellow boxes below is what I'm trying to confirm actuall happens.
75970
I am building a convolutional neural network to classify certain types of greyscale images (chess puzzles).  I can see obvious advantages for using ReLU as my activation function e.g. I can treat the input layer feed to the first hidden layer in the same way as the input into any other hidden layer in terms of the domain of the input.  But I have a question about how to use ReLU activations at the other end - i.e. as the output of the fully connected output layer.  I am looking to classify images of chess boards and so it seemed to me obvious that I could classify a part of the board as being a white square or a black square and then with some other (optional, as it were) classifications e.g. with pawn which itself could be black or white.  I cannot really see how to do this with ReLU - and so that suggests that what I really have to do is implement every possible outcome as a separate output neuron e.g., empty black square, empty white square, white square with white pawn, white square with black pawn, black square with white pawn ... etc.  Is that correct?  (I originally asked this on Stack Overflow and got no answers, but a recommendation to ask here. So I have deleted the original and posted this. Since I originally asked I have built a network and, using random weights - learning is about to begin, see that the overall results, after several layers of leaky ReLU activated filters and a leaky ReLU fully connected layer,  don't stray too far from the 0 - 1 range, though this may be luck more than anything else: but I am still intrigued by this problem so would welcome answers.)
667
What is the purpose of a neural network having a non-linear activation function?   Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?  I've read in other StackOverflow answers that the activation function "", but that is rather vague.  Another posting states that an answer in the context of , but again that is not exactly what I'm asking.
62060
I'm quite new to neural networks and currently I'm trying to train a non convolutional neural network on the MNIST data set. I'm observing some behaviour I don't quite understand.   This is the code written with keras as the library:  # the data, split between train and test sets (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()  y_train = keras.utils.to_categorical(y_train, 10) y_test = keras.utils.to_categorical(y_test, 10) model = Sequential()  early_stopper = EarlyStopping(patience=3)  model.add(Flatten(input_shape=(28,28))) model.add(Dense(128, activation="relu")) model.add(Dense(128, activation="relu")) model.add(Dense(10, activation="softmax")) model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])  model.fit(x_train, y_train, epochs=20, validation_data=(x_test, y_test), callbacks=[early_stopper])   This gives me a validation_acc of around 20%. The funny thing is, when I change the activation functions to "sigmoid", the loss function to "mean_squared_error" and the optimizer to "sgd" my performance improves to around 85% after 50 epochs.   Having read  I wonder what's the reason for the bad performance of the network I presented in the code. ReLU, cross-entropy and a dynamic optimizer like Adam all seem to improve on the idea of a very vanilla neural network with stochastic gradient optimization, mean squared error as loss and sigmoid activation functions. Yet I get a really bad performance and if I increase the number of nodes in the hidden layers I often get a network that doesn't learn at all.   EDIT: I figured out it has something to do with me not normalizing the input to values between 0 and 1 ... but why is this the problem?
78224
I am trying to understand biases in neural nets, but different websites show very different answers.   For example, how many biases is there in a fully connected neural network with a single input layer with 5 units and a single output layer with 4 units? And what about a fully connected neural network with a single input layer with 5 units, a single hidden layer with 4 units, and a single output layer with 3 units?  For example, if I understand this correctly, , the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias.  However, at , it is explained such that each layer including the input-layer has one bias. So the answer to the example above is one in the first and two in the second.   What is correct? What am I misunderstanding here?
27483
Background: I'm studying chapter 6 of Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville. In section 6.2.2.2 (pages 182 of 183 ) the use of sigmoid to output $P(y=1|x)$ is motivated.  To summarize some of the material they let $$z = w^Th+b$$ be an output neuron before an activation is applied where $h$ is the output of the previous hidden layer, $w$ is a vector of weights and $b$ is a scalar bias. The input vector is denoted $x$ (which $h$ is a function of) and the output value is denoted $y=\phi(z)$ where $\phi$ is the sigmoid function. The book wishes to deﬁne a probability distribution over $y$ using the value $z$. From the second paragraph of page 183:     We omit the dependence on $x$ for the moment to discuss how to deﬁne a   probability distribution over $y$ using the value $z$. The sigmoid can be motivated by constructing an unnormalized probability distribution $\tilde P(y)$, which does not sum to 1. We can then divide by an appropriate constant to obtain a valid probability distribution. If we begin with the assumption that the unnormalized log probabilities are linear in $y$ and $z$, we can exponentiate to obtain the unnormalized probabilities. We then normalize to see that this yields a Bernoulli distribution controlled by a sigmoidal transformation of z:   \begin{align} \log\tilde P(y) &amp;= yz \\ \tilde P(y) &amp;= \exp(yz) \\ P(y) &amp;= \frac{\exp(yz)}{\sum_{y'=0}^1 \exp(y'z) } \\ P(y) &amp;= \phi((2y-1)z) \end{align}   Questions: I'm confused about two things, particularly the first:   Where is the initial assumption coming from? Why is the unnormalized log probability linear in $y$ and $z$? Can someone give me some inituition on how the authors started with $\log\tilde P(y) = yz$? How does the last line follow?
78739
I've trained a neural network with two inputs, a single hidden layer with two neurons, and one output using a bipolar sigmoid activation function. If a single input is known, how would I determine the second input to create a desired output?  For example, let's say the neural network is trained to add two inputs to produce an output. So if input_1 = 3 and input_2 = 4, the output will be 7, (3 + 4 = 7). Given input_1 = 3 and the desired output is 7, I want to calculate the second input required to produce the desired output (the answer should be 4).  How would I do this for a network that is more complicated than basic addition and has multiple inputs/outputs? For example, for a network with four inputs and two outputs, how would I calculate input_3 and input_4 given input_1, input_2, output_1 and output_2?
63161
Whats a quick way to work the below (and problems similar) out?   For me to take this derivative it involves a lot of time and boring calculation, there has to be a better way.    This is taken from the solutions to Chris Bishop's Pattern Recognition and Machine learning:
67212
Suppose I have three layers: a foreground, an intermediate-ground and a background.  What is involved in using these three layers to create ?
27471
While reviewing the Transformer architecture, I realized something I didn't expect, which is that :   the positional encoding is summed to the word embeddings  rather than concatenated to it.               Based on the graphs I have seen wrt what the encoding looks like, that means that :   the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot,  while there is also a large amount of positions in the embedding that are only slightly affected by the positional encoding (when you move further towards the end).               So, why not instead have smaller word embeddings (reduce memory usage) and a smaller positional encoding retaining only the most important bits of the encoding, and instead of summing the positional encoding of words keep it concatenated to word embeddings?
47511
I recently read a lot about neural networks using GRU or LSTM units. There are many easy to use frameworks like tensorflow that do not even require high knowledge about programming. Unfortunately, I never really found good information on how the training of those networks work. Simple backpropagation might probably not work for gated recurrent networks or is just too inefficient for networks with such a high number of variables to learn.  So my question is:  What are the state-of-the-art algorithms used for the initialization and training of neural networks with GRU or LSTM units? I am not looking for frameworks to use, but for initialization and update equations for the internal parameters.
27437
How does the  for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
70301
* Note: The premise of my question was incorrect in the first place. My question assumes that an LSTM maintains a separate set of weights for each time step in the memory it is given as a design parameter. As , only a single set of weights is learned. So an LSTM cell does not care about how long a sequence is - it applies the same set of weights to each step in the sequence. Please don't let my question convolute your understanding of LSTMs.   Terminology:  Cell: the LSTM unit containing input, forget, output gates and the hidden hT and cell state cT. Hidden units/memory: How far back in time the LSTM is &quot;unrolled&quot;. A hidden unit is an instance of the cell at a particular time. A hidden unit is parameterized by [wT, cT, hT-1]: The gate weights for the current hidden unit, the current cell state, and the last hidden unit's output. Where wT represents input, output, forget gate weights.   An LSTM maintains separate gate weights wT for each hidden unit. This way it can treat different points in time of a sequence differently. (see *Note above) Let's say an LSTM has 3 hidden units so has gate weights w1, 2, w3 for each of them. Then a sequence x1, x2,...xN comes through. I am illustrating the cell as it transitions over time: @ t=1 xN....x3    x2    x1                   [w1, c1, h0]                       (c2, h1)  @ t=2 xN....x4    x3    x2                   x1                   [w2, c2, h1]                       (c3, h2)  @ t=3 xN....x5    x4    x3                   x2    x1                   [w3, c3, h2]                       (c4, h3)  But what happens at t=4? The LSTM only has memory, therefore gate weights, for 3 steps: @ t=4 xN....x6    x5    x4                   x3    x2    x1                   [w?, c3, h2]                       (c4, h3)  What weights are used for x4 and all the following inputs? In essence, how are sequences that are longer than an LSTM cell's memory treated? Do the gate weights reset back to w1, or do they remain static at their latest value wT?  Edit: My question is not a duplicate of  question. It is asking about multi-step prediction from inputs. However, I am asking about what weights are used over time for sequences that are longer than the internal hidden cell states. The question of weights is not addressed in that answer.
121214
An issue I've seen frequently brought up in the context of Neural Networks in general, and Deep Neural Networks in particular, is that they're "data hungry" - that is they don't perform well unless we have a large data set with which to train the network.  My understanding is that this is due to the fact that NNets, especially Deep NNets, have a large number of degrees of freedom. So as a model, a NNet has a very large number of parameters, and if the number of parameters of the model is large relative to the number of training data points, there is an increased tendency to over fit.   But why isn't this issue solved by regularization? As far as I know NNets can use L1 and L2 regularization and also have their own regularization methods like dropout which can reduce the number of parameters in the network.   Can we choose our regularizations methods such that they enforce parsimony and limit the size of the network?     To clarify my thinking: Say we are using a large Deep NNet to try to model our data, but the data set is small and could actually be modeled by a linear model. Then why don't the network weights converge in such a way that one neuron simulates the linear regression and all the others converge to zeros? Why doesn't regularization help with this?
22064
Suppose my training data contains ~100 variables, and each example is tagged as "success" or "failure".  I understand how a neural network can be used to try and predict success vs failure based on the variables.  However I am interested in the neural network outputting the posterior probability rather than success or failure. In fact, I evaluate the efficacy of the NN based on how accurate the probabilities are (eg. AUROC over the entire dataset) rather than % of cases with correct prediction.  Are NN's the right tool for the job here and if so how do you structure the NN to output this?  (NOTE: I'm a ML newbie!)