Can someone please elaborate on Gradient Descent vs Newton's Method? I am studying for my ML exam tomorrow and saw this as a bullet point but cannot find many great answers. Can anyone fill me in some details on the both of these, as deep as you would like to go.
87228
This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes:  After reviewing a set of lectures on convex optimization, Newton's method seems to be a far superior algorithm than gradient descent to find globally optimal solutions, because Newton's method can provide a guarantee for its solution, it's affine invariant, and most of all it converges in far fewer steps. Why is second-order optimization algorithms, such as Newton's method not as widely used as stochastic gradient descent in machine learning problems?
71640
I need some resources to get started on using neural networks for time series forecasting. I am wary of implementing some paper and then finding out that they have greatly over stated the potential of their methods. So if you have experience with the methods you are suggesting it will be even more awesome.
63592
Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient.  Let's assume we are building a model with ~10K parameters / weights. Is it possible to run the optimization using some gradient free optimization algorithms?   I think computing the numerical gradient would be too slow, but how about other methods such as Nelder-Mead, Simulated Annealing or a Genetic Algorithm?  All the algorithms would suffer from local minima, why obsessed with gradient?
63161
Whats a quick way to work the below (and problems similar) out?   For me to take this derivative it involves a lot of time and boring calculation, there has to be a better way.    This is taken from the solutions to Chris Bishop's Pattern Recognition and Machine learning:
83759
I have made two solvers to implement neural networks, one is based on stochastic gradient descent (SGD) while the other is based on the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm.   I have read a lot of material and find it is common to use SGD rather BFGS, but I have found that BFGS performs better than SGD.   Can anyone can tell me why people prefer SGD to BFGS?
89176
Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows? i.e. it is the same as 'normal' gradient descent, it's just the manner in which the training data is supplied that makes the difference?  One thing that confuses me is I've seen people say that even with SGD you can supply more than 1 data point, and have larger batches, so won't that just make it 'normal' mini-batch gradient descent?
79769
What is the meaning of "gradient with respect to $\mathbf x$"?    I am talking about the symbol $$\nabla_\mathbf x$$  Does that simply mean derivative with respect to $\mathbf x$?
17731
I'm reading , a post from the new distill journal. I'll paraphrase the main equations leading to the part which confuses me, the post describes the intuition in more detail.   The gradient descent algorithm is given by the following iterative process $$w^{k+1} = w^k-\alpha \nabla f(w^k)$$ where $w^k$ is the value of iteration $k$, the learning rate is $\alpha$ and $\nabla f(w)$ is  the gradient of the function $f$ evaluated at $w$. The function $f$ you wish to minimize.  Gradient descent with momentum is given by adding "memory" to the descent, this is described by the pair of equations:  \begin{align} z^{k+1} &amp;= \beta z^k + \nabla f(w^k) \\ w^{k+1} &amp;= w^k - \alpha z^{k+1}  \end{align}  In the next section "First Steps: Gradient Descent" the author considers a convex quadratic function  $$f(w) = \frac12w^TAw-b^Tw, \quad w \in \mathbb{R}^n, A \in \mathbb{R}^{n,n}$$ which has gradient $$\nabla f(w) = Aw-b$$ If we assume $A$ is symmetric and invertable then $f$ has optimal solution $w^\star = A^{-1}b$.   If we were to use gradient descent then we would iterate towards this optimal solution in the following way  \begin{align} w^{k+1} &amp;= w^k - \alpha \nabla f(w) \\ &amp;= w^k - \alpha (Aw^k -b) \end{align}  Then the article goes on to say "There is a very natural space to view gradient descent where all the dimensions act independently — the eigenvectors of $A$". I think this makes sense, although my intuition is kind of fuzzy.   Every symmetric matrix $A$ has an eigenvalue decomposition where $$A = Q\text{diag}(\lambda_1,\ldots,\lambda_n)Q^T.$$  Where $\lambda_1 &gt; \ldots &gt; \lambda_n$ and $Q$ is the vector with the corresponding eigenvectors as columns (right?).   This next part is where I don't understand what is going on:      If we perform a change of basis, $x^k = Q^T(w^k - w^\star)$, the   iterations break apart, becoming:       \begin{align} x_i^{k+1} &amp;= x_i^k - \alpha \lambda_i x_i^k  \\ &amp;=(1-\alpha\lambda_i)x_i^k &amp;= (1- \alpha\lambda_i)^{k+1}x_i^0 \end{align}      Moving back to our original space $w$, we can see that      $$w^k - w^\star = Qx^k = \sum\limits_{i}^n = x_i^0(1-\alpha\lambda_i)^kq_i$$   What is going on here? Where is the motivation of taking $w^k - w^\star$ into the eigendomain? What is $x^k$? Why are we now looking at invidual elements of the vector? I tried to follow the caculations through, but $x^{k+1}$ depends on $w^{k+1}$ which depends on $z^k$, which I thought we were trying to eliminate. My question is can someone expand on these few steps with some intuition and calculations? Thanks.
71006
For a minimization problem of function $f(x,y)$, does the equation $min_{x,y} f(x,y)=min_{y}min_{x}f(x,y)=min_{x}min_{y}f(x,y)$ always hold?  I saw this equation somewhere. It does the minimization in a sort of iterated way, treating either $x$ or $y$ as a constant first. But what about a saddle point? If I use Hessian matrix to consider the convexity of $f(x,y)$, it will let me know it doesn't work if there's a saddle point, while this iterated method may still give me a value.  I think this kind of problems falls into optimization, and I also wonder if  $min_{y}max_{x}f(x,y)=max_{x}min_{y}f(x,y)$ holds.
1877
I am a math master student and have done fundamental math courses like probability theory, measure theory, linear algebra and know a little bit about functional analysis. What is good way for me to learn machine learning in depth?  I have read the classical text  last summer; my impression was that it was very ineffective to read the book chapter by chapter like a mathematical text. The book does not go deep enough for many algorithms and skip too many steps considered too technical by engineers.   Is there a machine learning book that maybe does not cover too many topics, but treat each one in depth and takes advantage of math when necessary? It will be great to be able connect fundamental mathematical objects with machine learning (I am thinking about Lp spaces, hilbert space etc).
78659
What is the difference between the algorithms EM (Expectation Maximization) and Gradient Ascent (or descent)? Is there any condition under which they are equivalent?
91379
I was reading this paper () describing the coordinate descent algorithm for LASSO, and I can't quite figure out how the soft-thresholding update for each $\beta_j$ is derived for the linear regression case. Specifically, how does one go from equation (4) to equation (5) in the paper? Can anyone show me a detailed derivation of the update? Thanks!  Edit: actually this paper deals with the elastic net penalizing term, which reduces to LASSO when $\alpha=1$. I am mostly interested in LASSO, but I guess the derivation should be principally the same for elastic net.
40192
I am a bit confused about the derivation of MLE of Uniform$(0,\theta)$.  I understand that $L(\theta)={\theta}^{-n}$ is a decreasing function and to find the MLE we want to maximize the likelihood function.  What is confusing me is that if a function is decreasing, then wouldn't the function be maximized at the smallest input rather than the largest?  Thank you in advance for your help.
107264
When using support vector machine, are there any guidelines on choosing linear kernel vs. nonlinear kernel, like RBF? I once heard that non-linear kernel tends not to perform well once the number of features is large. Are there any references on this issue?
11359
I found rectified linear unit (ReLU) praised at several places as a solution to the  for neural networks. That is, one uses max(0,x) as activation function. When the activation is positive, it is obvious that this is better than, say, the sigmoid activation function, since its derivation is always 1 instead of an arbitrarily small value for large x. On the other hand, the derivation is exactly 0 when x is smaller than 0. In the worst case, when a unit is never activated, the weights for this unit would also never change anymore, and the unit would be forever useless - which seems much worse than even vanishingly small gradients. How do learning algorithms deal with that problem when they use ReLU?
65793
I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear.  It's reasonable that one differentiates between logical and linear type functions, depending on the desired binary/continuous output but what is the advantage of sigmoid function over the simple linear one?  ReLU is especially difficult to understand for me, for instance: what is the point to use a function that behaves like linear in case of positive inputs but is "flat" in case of negatives? What is the intuition behind this? Or is it just a simple trial-error thing, nothing more?
86995
In my machine learning class these two methods were discussed and mentioned that both should be used. I have a couple questions about this:  1) Can I mix and match these two approaches? e.g. Feature Scale x1 and Mean Normalize x2?  2) How do you determine which of these options to apply? It seems that either could accomplish the task of increasing your convergence rates... I suppose you just need to know your data set to understand which will reliably reduce your values while leaving as few outliers as possible?