I'm reading  and I have a problem with a piece of math in part related to gradient descent analysis.   Article's author analyzes the gradient descent on the simplest nontrivial problem: quadratic form with $A$ being symmetric and invertible: $$ f(w) = \frac{1}{2} w^TAw - b^Tw, \quad w \in \mathbb{R}^n. $$ The problem has optimal solution at $w^* = A^{-1}b$. We approach it though using gradient descent: $$ w^{k+1} = w^k - \alpha (Aw^k - b). $$ If we change the basis for $w^k = Qx^k + w^*$, where $Q$ is the matrix of $A$'s eigenvectors ($A = Q \Lambda Q^T$), we soon realize that: $$ w^k - w^* = Qx^k = \sum_i^n x_i^0 (1 - \alpha \lambda_i)^k q_i. $$ So we get the error at $k$-th iteration of gradient descent.  Then, in Decomposing the Error section, the author visualizes how error components contribute to the loss. To do so he writes the contributions of each eigenspace's error to the loss: $$ f(w^k) - f(w^*) = \sum(1 - \alpha \lambda_i)^{2k} \lambda_i [x_i^0]^2. $$ How did he get that? I tried using least squares, so I assumed $d=w^k - w^*$ and took the norm ($d^Td)$ but I did get: $$ \sum(1 - \alpha \lambda_i)^{2k} [x_i^0]^2 [q_i]^2. $$ Do I even approach it in the right way?
17731
I'm reading , a post from the new distill journal. I'll paraphrase the main equations leading to the part which confuses me, the post describes the intuition in more detail.   The gradient descent algorithm is given by the following iterative process $$w^{k+1} = w^k-\alpha \nabla f(w^k)$$ where $w^k$ is the value of iteration $k$, the learning rate is $\alpha$ and $\nabla f(w)$ is  the gradient of the function $f$ evaluated at $w$. The function $f$ you wish to minimize.  Gradient descent with momentum is given by adding "memory" to the descent, this is described by the pair of equations:  \begin{align} z^{k+1} &amp;= \beta z^k + \nabla f(w^k) \\ w^{k+1} &amp;= w^k - \alpha z^{k+1}  \end{align}  In the next section "First Steps: Gradient Descent" the author considers a convex quadratic function  $$f(w) = \frac12w^TAw-b^Tw, \quad w \in \mathbb{R}^n, A \in \mathbb{R}^{n,n}$$ which has gradient $$\nabla f(w) = Aw-b$$ If we assume $A$ is symmetric and invertable then $f$ has optimal solution $w^\star = A^{-1}b$.   If we were to use gradient descent then we would iterate towards this optimal solution in the following way  \begin{align} w^{k+1} &amp;= w^k - \alpha \nabla f(w) \\ &amp;= w^k - \alpha (Aw^k -b) \end{align}  Then the article goes on to say "There is a very natural space to view gradient descent where all the dimensions act independently — the eigenvectors of $A$". I think this makes sense, although my intuition is kind of fuzzy.   Every symmetric matrix $A$ has an eigenvalue decomposition where $$A = Q\text{diag}(\lambda_1,\ldots,\lambda_n)Q^T.$$  Where $\lambda_1 &gt; \ldots &gt; \lambda_n$ and $Q$ is the vector with the corresponding eigenvectors as columns (right?).   This next part is where I don't understand what is going on:      If we perform a change of basis, $x^k = Q^T(w^k - w^\star)$, the   iterations break apart, becoming:       \begin{align} x_i^{k+1} &amp;= x_i^k - \alpha \lambda_i x_i^k  \\ &amp;=(1-\alpha\lambda_i)x_i^k &amp;= (1- \alpha\lambda_i)^{k+1}x_i^0 \end{align}      Moving back to our original space $w$, we can see that      $$w^k - w^\star = Qx^k = \sum\limits_{i}^n = x_i^0(1-\alpha\lambda_i)^kq_i$$   What is going on here? Where is the motivation of taking $w^k - w^\star$ into the eigendomain? What is $x^k$? Why are we now looking at invidual elements of the vector? I tried to follow the caculations through, but $x^{k+1}$ depends on $w^{k+1}$ which depends on $z^k$, which I thought we were trying to eliminate. My question is can someone expand on these few steps with some intuition and calculations? Thanks.
29812
Figure below shows the accuracy using different alpha values in L2 regularisation. As long as alpha is small in the range of $10^{-12}$ to $10^{-2}$ the accuracy remain the same. I do undarstand when alpha value is $10^{1}$ or greater it will increase the weights to a point where they do not fit the data optimal and then, resulting in under-fitting. What is the reason why the accuracy remain the same by smaller alpha values? The formula is: $$w^{2}=w^{1}-\alpha*w^{1}$$  where $w^{1}$ and $w^{2}$ are respectively the recent and the regularised weight parameters, $\alpha$ is the regularisation parameter specifying the amount of regularisation.
86114
So I have this problem (self learning linear algebra) and I try to solve it using the indicated steps. Question is: Am I right, is this a valid answer? Obviously for me makes sense but since I'm just starting with maths maybe there's something wrong:     If $\|\vec{v}\| = 5$ and $\|\vec{w}\| =3$, what are the smallest and largest values of $\|\vec{v}-\vec{w}\|$ and $\vec{v} \cdotp \vec{w}$?   EDIT   I'd moved my initial answer to a full answer. See below.
63236
This is a homework problem, but I feel like I'm struggling with not knowing facts from linear algebra.  Apparently this is supposed to be an easy question but I hit a brick wall at the following point.  Can anyone give me a tip or tell me if I'm going down the wrong road?  My approach:  Let $A$ be a symmetric $n\times n$ matrix, and then suppose $x$ is a real-valued column vector of dimension $n$ with not all entries equal to $0$.  I need to show that $x^Te^{A}x&gt; 0$.  (based on looking up the definition of positive definite on wikipedia)  Then   $\begin{eqnarray*} x^{T}(e^{A})x &amp;=&amp; x^{T}(\sum_{n=0}^{\infty}\frac{A^{n}}{n!})x\\ &amp;=&amp; \sum_{n=0}^{\infty}\frac{x^{T}A^{n}x}{n!}\\ &amp;=&amp; \sum_{n=0}^{\infty}\frac{x^{T}A^{n}x}{n!})\\ \end{eqnarray*}$  As noted below, from here on is incorrect:  $\begin{eqnarray*} &amp;=&amp; \sum_{n=0}^{\infty}\frac{(x^{T}Ax)^{n}}{n!\|x\|^{2(n-1)}})\\ &amp;=&amp; \sum_{n=0}^{\infty}\frac{(x^{T}A^{T}x)^{n}}{n!\|x\|^{2(n-1)}}\\ &amp;=&amp; \sum_{n=0}^{\infty}\frac{((Ax)^{T}x)^{n}}{n!\|x\|^{2(n-1)}}\\ &amp;=&amp; \sum_{n=0}^{\infty}\frac{((Ax)^{T}(x^{T})^{T})^{n}}{n!\|x\|^{2(n-1)}}\\ &amp;=&amp; \sum_{n=0}^{\infty}\frac{((x^{T}Ax)^{T})^{n}}{n!\|x\|^{2(n-1)}}\\ \end{eqnarray*}$  As you can see this brings me no closer to getting $x^Tx$ somewhere, which I may assume is greater than $0$.  So my conclusion is that using only the fact that $A$ is symmetric is not enough.  Is there some result about symmetric matrices that I should use?
73007
I know I mess up different things. However I want to get better understanding what is motivation behind logistic regression.  $p(y_i=1|x_i)=(1+e^{-x_iw})^{-1}$  and according to OLS  $S(w)=\sum_{i}^{}(y_i-\hat{y_i})^2$,where $y \in \{1,0\}$.  What if $\hat{y_i}=p(y_i=1|x_i)$ and estimate $w$ by finding minimum to $S(w)$.  Why this is incorrect approach?
88831
Suppose I am given a $2$x$2$ matrix $A=$ \begin{pmatrix} a &amp; b \\ c &amp; d \end{pmatrix}  And an initial vector $x_n$ = \begin{pmatrix} x_0 \\ y_0\end{pmatrix}.  Under repeated iteration $x_{n+1} = Ax_n$, we find that $x_{n+1}$ converges to a line in the direction of the $eigenvector$  of $A$ of the form $X$= $[u,mu]$ where $m$ is the slope of the line.  I know that this process somehow also involves the largest $eigenvalue, \lambda,$ of $A$, though I am unsure how it connects.  My question is: How does the largest eigenvalue connect to this repeated iteration, how do I find the slope of the line of convergence $m$, and what is meant by the "direction of the eigenvector"?
118650
The question is very simple: why, when we try to fit a model to our data, linear or non-linear, do we usually try to minimize the sum of the squares of errors to obtain our estimator for the model parameter? Why not choose some other objective function to minimize? I understand that, for technical reasons, the quadratic function is nicer than some other functions, e.g., sum of absolute deviation. But this is still not a very convincing answer. Other than this technical reason, why in particular are people in favor of this 'Euclidean type' of distance function? Is there a specific meaning or interpretation for that?  The logic behind my thinking is the following:  When you have a dataset, you first set up your model by making a set of functional or distributional assumptions (say, some moment condition but not the entire distribution). In your model, there are some parameters (assume it is a parametric model), then you need to find a way to consistently estimate these parameters and hopefully, your estimator will have low variance and some other nice properties. Whether you minimize the SSE or LAD or some other objective function, I think they are just different methods to get a consistent estimator. Following this logic, I thought people use least square must be 1) it produces consistent estimator of the model 2) something else that I don't know.   In econometrics, we know that in linear regression model, if you assume the error terms have 0 mean conditioning on the predictors and homoscedasticity and errors are uncorrelated with each other, then minimizing the sum of square error will give you a CONSISTENT estimator of your model parameters and by the Gauss-Markov theorem, this estimator is BLUE. So this would suggest that if you choose to minimize some other objective function that is not the SSE, then there is no guarantee that you will get a consistent estimator of your model parameter. Is my understanding correct? If it is correct, then minimizing SSE rather than some other objective function can be justified by consistency, which is acceptable, in fact, better than saying the quadratic function is nicer.   In pratice, I actually saw many cases where people directly minimize the sum of square errors without first clearly specifying the complete model, e.g., the distributional assumptions (moment assumptions) on the error term. Then this seems to me that the user of this method just wants to see how close the data fit the 'model' (I use quotation mark since the model assumptions are probably incomplete) in terms of the square distance function.  A related question (also related to this website) is: why, when we try to compare different models using cross-validation, do we again use the SSE as the judgment criterion? i.e., choose the model that has the least SSE? Why not another criterion?
71439
Given the vectors $x_0, \theta \in \mathbb{R}^n$, $\varepsilon\in \mathbb{R}^+$, and symmetric positive definite matrices $A, \Sigma\in \mathbb{R}^{n\times n}$, how the following optimization problem can be solved (possibly in closed form)? $$\begin{array}{ll} \underset{x \in \mathbb{R}^n}{\text{minimize}} &amp; (x - x_0)^\top A (x - x_0)\\ \text{subject to} &amp; \sqrt{x^\top \Sigma x} + x^\top \theta \leq \varepsilon\end{array}$$ What I tried: First, note that the problem is convex since both the objective is convex in $x$, and the constraint defines a convex set in $x$. Write the Lagrangian of the problem as $$L(x,\lambda) = (x-x_0)^\top A (x-x_0) + \lambda (\sqrt{x^\top \Sigma x} + x^\top \theta  - \varepsilon). $$ The KKT conditions for this problem leads to the system: $$ \begin{cases} A (x-x_0) + \lambda \left(\frac{\Sigma x}{\sqrt{x^\top\Sigma x}}+ \theta \right) = 0 \\ \lambda\left(\sqrt{x^\top \Sigma x} + x^\top \theta  - \varepsilon \right) = 0 \\ \sqrt{x^\top \Sigma x} + x^\top \theta  - \varepsilon \leq 0\\\lambda\geq 0 \end{cases} $$ However, the solution of this system gets complicated in my calculation and I am not able to find a closed-form solution. Any suggestion?
2545
(a) Given the column vector $a$ as an $n\times 1$ matrix. Write out its singular value decomposition, showing the matrices $U, \sum$, and $V$ explicitly.  (b) Given the row vector $a^T$ as an $1\times n$ matrix. Write out its singular value decomposition, showing the matrices $U, \sum$, and $V$ explicitly.  I currently get stucked on this question, since I couldn't find the eigenvalues of the $n\times n$ matrix $aa^T$ in part (a), and $a^Ta$ in part (b) to find the columns of $U$ and $V$, respectively. Can someone please help me on this problem?
437
I got a slight confusion on the  algorithm used in  (MLP).  The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of the hidden layers. The output error I can understand, that is, e = d - y [Without the subscripts].  The questions are:   How does one get the error of hidden layer? How does one calculate it? If I backpropagate it, should I use it as a cost function of an adaptive filter or should I use a pointer (in C/C++) programming sense, to update the weight?
117473
Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited.  I am trying to train a simple neural network to learn a simple quadratic function of the form: $f(x) = 5 - 3x + 2x^2$  I set up a single-layered network with a single neuron. The input is a 2d array of the form $(x, x^2)$ and I don't use an activation function. I expect that the weights and biases I extract from the network will correspond to the coefficients in the function $f(x)$.  I randomly generate some training points and labels, as well as a validation data set, and train my model using the Keras sequential model called from TensorFlow.   import tensorflow as tf from tensorflow import keras import numpy as np import matplotlib.pyplot as plt  def fTest(x_arg):     return 5 - 3*x_arg + 2*(x_arg)**2   # training data t = np.random.choice(np.arange(-10,10, .01),5000 ) t1 = [] for i in range(len(t)):     t1.append([t[i], t[i]**2]) s = [] for i in range(len(t)):     s.append(fTest(t[i])) t1 = np.array(t1) s = np.array(s)  # validation set v = np.random.choice(np.arange(-10,10, .01),5000 ) v1 = [] for i in range(len(v)):     v1.append([v[i], v[i]**2]) u = [] for i in range(len(v)):     u.append(fTest(v[i])) v1 = np.array(v1) u = np.array(u)  model = keras.Sequential([     keras.layers.Dense(1, input_shape=(2,) , use_bias=True), ])  model.compile(optimizer='adam',                loss='mean_squared_logarithmic_error',               metrics=['mae','accuracy'])  model.fit(t1, s, batch_size=50, epochs=2000, validation_data=(v1,u))    The model seems to train, but very poorly. The 'accuracy' metric is also zero, which I am very confused about.   Epoch 2000/2000 200/200 [==============================] - 0s 23us/step - loss: 0.0018 - mean_absolute_error: 1.0144 - acc: 0.0000e+00 - val_loss: 0.0014 - val_mean_absolute_error: 1.0276 - val_acc: 0.0000e+00   Visually, the predictions of the model seem to be reasonably accurate    I've tried other loss-functions but none of them seem to work any better. I'm fairly new to to using TF/Keras so is there something obvious that I'm missing?  Edit: corrected training output
90523
I was following the backpropagation tutorial by Michael Nielsen on , one of the very few places where the backprop algorithm is nicely explained both in theory and practice with an actual implementation in python.  Now, something is not clear when he explains how to calculate the error on a specific layer in terms of error in the next layer (I will provide a screenshot because I don't want to mess with notation):    Here, delta^l is the error in layer l, the circled dot is the elementwise product betweeen vectors, w is a weight matrix, and sigma primed indicates the derivative of an activation function.  Now, observe he points out that by applying the transpose operator to the next-layer-error vector (which is the previous layer during backprop), one is "sending back" the error across the network, and then, multiplying by the derivative of sigma, one applies the chain rule which completes the backprop step.  I have troubles understanding what does he really mean by saying that the traspose operator "sends back".  Perhaps he wants to apply the inverse trasformation? But unless the matrix happens to be orthogonal, which is not the case AFAIK, the transpose does NOT equal the inverse..  In any case, I would have thought of applying some kind of differential operator to matrix w, and not the transpose one.  Opinions?  NOTE: Please let me know whether this post is correctly suited for stackoverflow, or should it be posted on stats or math stackexchange.
26711
I was taking an online course and saw linear regression being by gradient descent The intuition behind why the method would work seemed plausible.  I tried understanding normal equation as to why setting individual partial derivates to 0 and solving the equations give the optimal values of theta, but that didn't ring a bell.   Why setting partial derivatives to zero, and solving the equations gives optimal value of theta. I also went through the following . The part about The minimum is determined by calculating the partial derivatives of S(β1,β2) with respect to β1 and β2 and setting them to zero is still dicey to me. Why would it work. Why would it give the values of β1 and β2 for minimized cost function. If we have multiple local optima, would the normal equation method give the global optima? If so, why?
102687
Background  Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$  where $\mathbf{\beta}$ is an $(k\times1)$ vector of coefficients, $\mathbf{X}$ is the design matrix defined by  $$\mathbf{X} = \begin{pmatrix} 1 &amp; x_{11} &amp; x_{12} &amp; \dots &amp; x_{1\;(k-1)} \\ 1 &amp; x_{21} &amp; \dots  &amp;       &amp;   \vdots \\ \vdots &amp;   &amp; \ddots &amp;       &amp;   \vdots \\ 1 &amp; x_{n1} &amp; \dots  &amp; \dots &amp;   x_{n\;(k-1)}     \end{pmatrix}$$ and the errors are IID normal,  $$\mathbf{\epsilon} \sim \mathcal{N}\left(\mathbf{0},\sigma^2 \mathbf{I}\right) \;.$$  We minimize the sum-of-squared-errors by setting our estimates for $\mathbf{\beta}$ to be $$\mathbf{\hat{\beta}}= (\mathbf{X^T X})^{-1}\mathbf{X}^T \mathbf{y}\;. $$   An unbiased estimator of $\sigma^2$ is  $$s^2 = \frac{\left\Vert \mathbf{y}-\mathbf{\hat{y}}\right\Vert ^2}{n-p}$$ where $\mathbf{\hat{y}} \equiv \mathbf{X} \mathbf{\hat{\beta}}$ ().   The covariance of $\mathbf{\hat{\beta}}$ is given by $$\operatorname{Cov}\left(\mathbf{\hat{\beta}}\right) = \sigma^2 \mathbf{C}$$ where $\mathbf{C}\equiv(\mathbf{X}^T\mathbf{X})^{-1}$ () .  Question  How can I prove that for $\hat\beta_i$, $$\frac{\hat{\beta}_i - \beta_i} {s_{\hat{\beta}_i}} \sim t_{n-k}$$ where $t_{n-k}$ is a t-distribution with $(n-k)$ degrees of freedom, and the standard error of $\hat{\beta}_i$ is estimated by $s_{\hat{\beta}_i} = s\sqrt{c_{ii}}$.    My attempts  I know that for $n$ random variables sampled from $x\sim\mathcal{N}\left(\mu, \sigma^2\right)$, you can show that  $$\frac{\bar{x}-\mu}{s/\sqrt{n}} \sim t_{n-1} $$ by rewriting the LHS as $$\frac{ \left(\frac{\bar x - \mu}{\sigma/\sqrt{n}}\right)  }   {\sqrt{s^2/\sigma^2}}$$ and realizing that the numertor is a standard normal distribution, and the denominator is square root of a Chi-square distribution with df=(n-1) and divided by (n-1) ().  And therefore it follows a t-distribution with df=(n-1) ().  I was unable to extend this proof to my question...  Any ideas?  I'm aware of , but they don't explicitly prove it, they just give a rule of thumb, saying "each predictor costs you a degree of freedom".
4688
It's stated that the gradient of:  $$\frac{1}{2}x^TAx - b^Tx +c$$  is   $$\frac{1}{2}A^Tx + \frac{1}{2}Ax - b$$  How do you grind out this equation? Or specifically, how do you get from $x^TAx$ to $A^Tx + Ax$?
27027
Given a matrix $\mathbf V^{m \times n}$,  (NMF) finds two non-negative matrices $\mathbf W^{m \times k}$ and $\mathbf H^{k \times n}$ (i.e. with all elements $\ge 0$) to represent the decomposed matrix as:  $$\mathbf V \approx \mathbf W\mathbf H,$$  for example by requiring that non-negative $\mathbf W$ and $\mathbf H$ minimize the reconstruction error $$\|\mathbf V-\mathbf W\mathbf H\|^2.$$  Are there common practices to estimate the number $k$ in NMF? How could, for example, cross validation be used for that purpose?
49456
Define $$L(w,u)=\frac{1}{2}\|w-u\|^2+\beta \left\|\frac{w}{x}\right\|,~w,u\in \Bbb{R}^n$$ where $$\frac{w}{x}=\left(\frac{w_1}{x_1},\ldots, \frac{w_n}{x_n}\right)$$ $$\|x\|=\sqrt{x_1^2+\cdots+x_n^2}$$ Given $u$, $x$ and $\beta$, how to get $$\arg\min_{w\in \Bbb{R}^n}L(w,u)$$