The universal approximation theorem says that a neural network can be used to approximate any continuous function under some regular conditions with arbitrarily small approximation error, provided we have enough number of nodes in the hidden layer.  Suppose a response y depend on a p-dimensional vector x only through s of the entries (s  However, in practice, this is not usually the case. Why neural network can not learn the sparse structure automatically? Is it because of the non-convexity? Or we don't have enough sample size to learn the specific relationship?
121214
An issue I've seen frequently brought up in the context of Neural Networks in general, and Deep Neural Networks in particular, is that they're "data hungry" - that is they don't perform well unless we have a large data set with which to train the network.  My understanding is that this is due to the fact that NNets, especially Deep NNets, have a large number of degrees of freedom. So as a model, a NNet has a very large number of parameters, and if the number of parameters of the model is large relative to the number of training data points, there is an increased tendency to over fit.   But why isn't this issue solved by regularization? As far as I know NNets can use L1 and L2 regularization and also have their own regularization methods like dropout which can reduce the number of parameters in the network.   Can we choose our regularizations methods such that they enforce parsimony and limit the size of the network?     To clarify my thinking: Say we are using a large Deep NNet to try to model our data, but the data set is small and could actually be modeled by a linear model. Then why don't the network weights converge in such a way that one neuron simulates the linear regression and all the others converge to zeros? Why doesn't regularization help with this?
123820
I am a sophomore student who's interested in deep learning and its method layering up some linear/non-linear operations and constructs up the complex function through the network.  I'd like to comprise a network that finds the word upon the given explanations :  e.g.1 input: a spherically shaped fruit which tastes sour and sweet and gets red when matured well output: apple  e.g.2, input: to make up the boundary between multiple options for the base of one's behavior or decision output: determine.  So basically it's a classification problem but the class might be more than 20,000.  I'd like to name this task "rephrase neural network".  Any good reference or starting point, or, any network recommendation for comprising this neural network?
59890
How does a  work, and what differentiates it from other linear classifiers, such as the , , or ? *  (* I'm thinking in terms of the underlying motivations for the algorithm, optimisation strategies, generalisation capabilities, and run-time complexity)
117473
Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited.  I am trying to train a simple neural network to learn a simple quadratic function of the form: $f(x) = 5 - 3x + 2x^2$  I set up a single-layered network with a single neuron. The input is a 2d array of the form $(x, x^2)$ and I don't use an activation function. I expect that the weights and biases I extract from the network will correspond to the coefficients in the function $f(x)$.  I randomly generate some training points and labels, as well as a validation data set, and train my model using the Keras sequential model called from TensorFlow.   import tensorflow as tf from tensorflow import keras import numpy as np import matplotlib.pyplot as plt  def fTest(x_arg):     return 5 - 3*x_arg + 2*(x_arg)**2   # training data t = np.random.choice(np.arange(-10,10, .01),5000 ) t1 = [] for i in range(len(t)):     t1.append([t[i], t[i]**2]) s = [] for i in range(len(t)):     s.append(fTest(t[i])) t1 = np.array(t1) s = np.array(s)  # validation set v = np.random.choice(np.arange(-10,10, .01),5000 ) v1 = [] for i in range(len(v)):     v1.append([v[i], v[i]**2]) u = [] for i in range(len(v)):     u.append(fTest(v[i])) v1 = np.array(v1) u = np.array(u)  model = keras.Sequential([     keras.layers.Dense(1, input_shape=(2,) , use_bias=True), ])  model.compile(optimizer='adam',                loss='mean_squared_logarithmic_error',               metrics=['mae','accuracy'])  model.fit(t1, s, batch_size=50, epochs=2000, validation_data=(v1,u))    The model seems to train, but very poorly. The 'accuracy' metric is also zero, which I am very confused about.   Epoch 2000/2000 200/200 [==============================] - 0s 23us/step - loss: 0.0018 - mean_absolute_error: 1.0144 - acc: 0.0000e+00 - val_loss: 0.0014 - val_mean_absolute_error: 1.0276 - val_acc: 0.0000e+00   Visually, the predictions of the model seem to be reasonably accurate    I've tried other loss-functions but none of them seem to work any better. I'm fairly new to to using TF/Keras so is there something obvious that I'm missing?  Edit: corrected training output
69999
When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to slowly locate the minimum point with respect to a certain weight, why don't we just set the derivative $\frac{d(\text{Error})}{dw}=0$, and find the value of weight $w$ which minimizes the error? Also, why are we sure that the error function in back-propagation will be a minimum? Can't it turn out the error function is a maximum instead? Is there a specific property of the squashing functions that guarantees that a network with any number of hidden nodes with arbitrary weights and input vectors will always give an error function that has some minima?
68819
Reading the comment to  the user makes the argument that one can pick a dense subset of a separable vector spaces without relying the axiom of countable choices. Since the comment got a lot of up votes, I guess it's correct, but to me this seems very strange. My rule of thumb was/is the following: We can get by without a choice function if   There is a canonical choice at each iteration. The proof doesn't actually require the iteration process to terminate.   None of these seem to be satisfied in the present case, unless there is a canonical choice for the dense subset, or the set is dense at some finite iteration, so my question is:   When can I tell if an induction arguments relies on the existence of a countable choice function?
89695
I am using feed-forward NN. I understand the concept, but my question is about weights. How can you interpret them, i.e. what do they represent or how can they be undestrood (besied just function coefficients)? I have found something called "space of weights", but I am not quite sure what does it means.
45382
I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).  Why is this a good idea?
35720
I am reading the Pattern Recognition and Machine Learning book by Bishops. I have some difficulties proving the (6.21) equation. Can someone help me?   N.B: If someone has a better title for this question, i'm opened to it as i couldn't find the good words.
38514
Is there a way or theorem that allows me to constrain a function $N(x)$, where $N(x)$ is a feedforward mlp such that it will always be a monotonic function. The simplest way is of course to use a penalty method, but this does not 100% guarantee, I am looking to see if there is a function or transformation that can be applied to $N(x)$ that constrains its form to being monotonic.  The only way I can currently think of is by making all weights $w_{i,j,k} &gt;0$ and that the activation function $f(a)$ are monotonic and non-negative (using $f(a) = \frac{tanh(a)+1}{2}$) and having a zero bias.
22337
The question is simple: is there any advantage in using sigmoid function in a convolutional neural network? Because every website that talks about CNN uses Relu function.
32456
I've got an equation that contains   $$x^p - 1$$   $x$ is any positive number (such as 2) and $p$ is a small positive number close to 0 (such as 0.001).  For some reason (that I may have known in High School), this can be approximated by   $$\ln(x) \times p$$  Can anyone explain why this approximation works?  Does it have something to do with approximating a natural log with a series?
66521
I realize that every inner product defined on a vector space can give rise to a norm on that space.  The converse, apparently is not true.  I'd like an example of a norm which no inner product can generate.
63161
Whats a quick way to work the below (and problems similar) out?   For me to take this derivative it involves a lot of time and boring calculation, there has to be a better way.    This is taken from the solutions to Chris Bishop's Pattern Recognition and Machine learning:
63480
I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of hyperparameters and I'm very confused on how to choose them. Also using cross-validation is not an option since training really takes a lot of time!
44974
It's quite intuitive that most neural network topologies/architectures are not identifiable. But what are some well-known results in the field? Are there simple conditions which allow/prevent identifiability? For example,    all networks with nonlinear activation functions and more than one hidden layer are not identifiable all networks with more than two hidden units are not identifiable   Or things like these. NOTE:I'm not saying that these conditions prevent identifiability (though they seem pretty good candidates to me). They are just examples of what I mean with "simple conditions".  If it helps to narrow down the question, feel free to consider only feed-forward and recurrent architectures. If this is still not enough, I'd be satisfied with an answer which cover at least one architecture among MLP, CNN and RNN. I had a quick look around on the Web but it looks like the only discussion I could find was on Reddit. Come on, people, we can do better than Reddit ;-)
42147
In , it was discussed how LA is a foundation to other branches of mathematics, be they pure or applied. One answer argued that linear problems are fully understood, and hence a natural target to reduce pretty much anything to.   Now, it's evident enough that such a linearisation, if possible, tends to make hard things easier. Find a Hilbert space hidden in your domain, obtain an orthonormal basis, and bam, any point/state can be described as a mere sequence of numbers, any mapping boils down to a matrix; we have some good theorems for existence of inverses / eigenvectors/exponential objects / etc..  So, LA sure is convenient.  OTOH, it seems unlikely that any nontrivial mathematical system could ever be said to be thoroughly understood. Can't we always find new questions within any such framework that haven't been answered yet? I'm not firm enough with G&ouml;del's incompleteness theorems to judge whether they are relevant here. The first incompleteness theorem says that discrete disciplines like number theory can't be both complete and consistent. Surely this is all the more true for e.g. topology.  Is LA for some reason exempt from such arguments, or does it for some other reason deserve to be called the best understood branch of mathematics?