I'm aware there is a very relevant explanation on L1 regularization's effect on feature selection at here:  [Ref. 1].   To better understand it I'm reading Google's tutorial on  [Ref. 2].  When it comes to the following part, there's some statements I emphasized that I do not understand:     You can think of the derivative of L1 as a force that subtracts some   constant from the weight every time. However, thanks to absolute   values, L1 has a discontinuity at 0, which causes subtraction results   that cross 0 to become zeroed out. For example, if subtraction   would have forced a weight from +0.1 to -0.2, L1 will set the weight   to exactly 0. Eureka, L1 zeroed out the weight.   I imagine when it says "L1 has a discontinuity at 0" it means the loss of the L1 like in the following  figure [Ref. 1]:    But why it will "cause subtraction results that cross 0 to become zeroed out"?  Why "if subtraction would have forced a weight from +0.1 to -0.2, L1 will set the weight to exactly 0"?  Is it related to that L1 is not differentiable at $w = 0$?
15973
I've a question regarding to the graphical intuition of the LASSO. I'm understanding why the lasso produce zero coefficient in case of intersecting a corner of the diamond. But I don't understand the case when the lasso Regression just shrink the paramters and don't set them to Zero. So if the RSS lines intersect with the "side" of the diamond. Assume in the case of the figure, that $\beta_1$ bigger and $\beta_2$ smaller. So the RSS-line would likely intersect a side.   Could you give me some intuition when the RSS-Line tangents with the side of the diamond instead of the corner?
53760
I am building a simple logistic regression model on 2D data. Here is the input I use.    I built a logistic regression model using this data and it successfully is able to find the discriminating line between these two classes. Here is the separating line found by my program:    Also, just for debugging, I plotted number of iterations vs J(theta), which also looks fine:    Now, confusing part comes into play. I am investigating the cost surface of this model that the logistic regressions is doing the gradient descent on. I am considering three different loss functions. Sum-of-squared loss, hinge-loss, log-loss.  Hinge loss:    Log-loss:    Sum-of-squared loss:    As far as I know, both hinge loss, sum-of-squared loss and log-loss should produce a convex surface. When I look at my surfaces, however, sum-of-squared surface does not look convex to me. Why is that the case?  For convenience, here is the two critical functions from my code:  function retval = J(X, y, theta)     M = size(y, 1);     % sum of squared error     retval = (1/(2*M)) * sum((h(X, theta) - y).^2);  endfunction  function retval = h (X, theta)     % logistic     retval =  1 ./ (1+ exp(-1 * (X * theta))); endfunction
60381
Notation: $\textbf{w}$ is an M-dimensional vector of parameters (including the bias parameter), $\textbf{x}_n$ is an M-dimensional vector of the features of each training example, $\textbf{t}$ is an N-dimensional vector of target values - where $t_n$ is the output of the nth training example and finally, $\Phi$ is an $M \times N$ design matrix.  During studying linear regression, I naturally came across the derivation of the normal equation. In his book, Christopher M. Bishop begins by the log of the likelihood function given by:  $$ \frac{N}{2} \ln \beta - \frac{N}{2} \ln(2\pi) - \frac{\beta}{2}\sum_{n=1}^N\{t_n - \textbf{w}^T \phi(\textbf{x}_n)\}^2 \tag{1} $$  With the goal of calculating the maximum likelihood of the parameters, he then calculates the derivative with respect to $w$:  $$ \bigtriangledown p(\textbf{t}| \textbf{w},\beta) = \beta \sum_{n=1}^N\{t_n - \textbf{w}^T \phi(\textbf{x}_n)\}\phi(\textbf{x}_n)^T \tag 2 $$  After which he sets the derivative to zero, in order to maximise the likelihood function  $$ 0 = \sum_{n=1}^N t_n \phi(\textbf{x}_n)^T - \textbf{w}^T \bigg(\sum_{n=1}^N \phi(\textbf{x}_n)\phi(\textbf{x}_n)^T \bigg) \tag 3 $$  So far so good. But then, the result for the maximum likelihood of the parameters $w$ is written as:  $$ \textbf{w}_{ML} = (\Phi^T\Phi)^{-1} \Phi^T \textbf{t} \tag 4 $$  My own derivation from equation $(3)$ to equation $(4)$ goes along the following lines:  \begin{align*} \Phi^T \Phi \textbf{w} &amp;= \Phi^T \textbf{t} \tag 5 \\  (\Phi^T \Phi)^{-1} \Phi^T \Phi \textbf{w} &amp;= (\Phi^T \Phi)^{-1} \Phi^T \textbf{t} \tag 6 \\ \textbf{w} &amp;= (\Phi^T \Phi)^{-1} \Phi^T \textbf{t} \tag 7 \end{align*}  My question is this: How does one go from equation $(3)$ to equation $(4)$. How does the supposed $ \textbf{w}^T \phi(\textbf{x}_n) \phi(\textbf{x}_n)$ become $\Phi^T \Phi \textbf{w}$ ? I do of course understand that given the dimensions of the matrix $\Phi$ and the vector $\textbf{w}$, that this is the only way to multiply the two entities. But what is the steps that one would follow in a formal derivation in order to reach such a result?
121856
For the input training set ${ \{ ({ x }_{ i }{ y }_{ i })\}  }_{ i=1 }^{ n }$  if the loss function is L(y, f(x)), then we initialize the model $M_0$ by finding the $\gamma$ which minimizes: $$ F_0(x) = \sum _{ i=1 }^{ n }{ L{ (y }_{ i, } } \gamma ) $$  which means, for every 'x' we define a model which always gives a constant value $\gamma$  So, now, in the 1st iteration, how come we are able to calculate the derivate of Loss function with respect to the previous model's function, (which is a constant $\gamma$), as derivates with respect to constant are not defined.  Can anyone explain what I'm understanding wrong over here ?
17284
I'm seeking your help in understanding the difference between two different contrasts for dichotomous variables.  On this page:  under "Dichotomous Predictor Variables", there are two ways to code dichotomous predictors: using the contrast 0,1 or the contrast 1,-1. I kind of understand the distinction here (0,1 is dummy coding and 1,-1 adds to one group and subtracts from the other) but don't understand which to use in my regression.  For example, if I have two dichotomous predictors, gender (m/f) and athlete (y/n), I could use contrasts 0,1 on both or 1,-1 on both. What would be the interpretation of a main effect or an interaction effect when using the two different contrasts? Does it depend on whether my cells are of different sizes?
40192
I am a bit confused about the derivation of MLE of Uniform$(0,\theta)$.  I understand that $L(\theta)={\theta}^{-n}$ is a decreasing function and to find the MLE we want to maximize the likelihood function.  What is confusing me is that if a function is decreasing, then wouldn't the function be maximized at the smallest input rather than the largest?  Thank you in advance for your help.
3668
I have a question regarding interpretation of regression results based on data where I have some negative values. Since the residuals were positively skewed I needed to log-transform my data, and because of negative values my log transformation looked like $\ln(Y-\min(Y)+1)$. Normally, if we have $\ln$ in both sides of the equation, then our $b$ coefficient means a percentage change of DV for 1 % change of IV. But I have big doubts whether the same is valid for such log-transformed data, where I added a constant to my DV in order to make the minimum value equal to 1. I can't also find any literature on this topic. Does anyone know for sure what would be the correct interpretation in such a situation?
21608
In  it says     To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good approximation. If we don't, we might end up with $\Delta C&gt;0$, which obviously would not be good! At the same time, we don't want $\eta$ to be too small, since that will make the changes $\Delta v$ tiny, and thus the gradient descent algorithm will work very slowly. In practical implementations, $\eta$ is often varied so that Equation (9) remains a good approximation, but the algorithm isn't too slow. We'll see later how this works.   But just a few paragraphs before we established that $\Delta C\approx−\eta\nabla C⋅\nabla C=−\eta\|\nabla C\|^2$ is obviously always negative (for positive $\eta$). So how can $\Delta C$ be positive if we don't choose a small enough learning rate? What is meant there?
76963
I'm currently learning about gradients, and I thought khanacademy could help me acquiring some intuition. The actual computation is clear to me, however I'm having trouble understand the intuition.    is the video I'm talking about, and the part I have a part with is 6:11. He says the gradient vector shows the direction you have to travel in the x,y-plane in order to get a maximum slope in the z-direction.  This sounds like gibberish to me. How do you have to 'travel' it? Why do you get a maximum slope? I just have no clue.
123333
These two are supposed to be equivalent because of the properties of logarithms, but the domains of $\ln(x^2)$ and $2\ln x$ seem different to me. For example, if I substitute $x=-1$ into the first, I get 0. But in the second, I get a non-real answer.   Why is this? The domains of these functions, when graphed, seem different as well. But everything I've been taught so far, and most of the things I can find on the web, do not explain this inconsistency. Typically, I consider the property $\ln(a^b) = b \ln a$ to be true...  I am only a senior in high school, currently in Calculus I, but hopefully the explanation won't be too outside of the realm of my understanding. But even if it is, I would still like to know the answer.   Thanks in advance
101080
hello I have a question about weight regularization in Adam apparently the weight_decay in the AdamW function  has the same impact as an L2 regularization my questions are: is that parameter the same as lambda that we have in the regularization term?  how does it exactly work? and what is its impact on the model complexity?
87424
I am minimizing a convex function $f(x,y)$ using the steepest descent method: $$\mathbf{x}_{n+1}=\mathbf{x}_n-\gamma \nabla F(\mathbf{x}_n),\ n \ge 0$$  My function is defined over a specific domain $D = \{(x, y) \in R^2 : 2x^2+y^2 &lt; 10\}$, if my $x_{n+1}$ goes out of bound, my method diverge. I was told to choose $\gamma$ using armijo rule to stay inbound.   Can someone explain to me the method and also describe a pseudo code for implementing it?
104108
Consider multiple linear regression. This question might be deceptively simple, but I'm trying to intuitively understand why, say if I have predictors X1 and X2, then interactions between these predictors can be adequately captured by X1 * X2.  I know interaction terms are modelled as products, just because that's what I was taught in school, and that's what everyone says to do.  I'm guessing there is maybe some geometric argument.  But why is a product (of say two numeric features, and not the extra complexity of multiplying by one being a dummy variable while other is numeric etc) going to adequately capture interactions?  Why are not "interactions" best captured by another f(X1, X2) by default instead of specifically X1 * X2?  I can see the idea that X1 * X2 may capture situations where the signs of X1 and X2 are the same or not, but then why would not, say, by default interactions be modelled by say f(X1, X2) = sign(X1) * sign(X2) instead of f(X1, X2) = X1X2?  I realise I can add any other f(X1, X2) to a regression or any predictive model, but finding the exact shape of interactions by hand coding is time consuming. How do I know X1X2 is a good first guess?
109322
This question pertains to L1 &amp; L2 regularization parameters in Light GBM. As per official documentation:  reg_alpha (float, optional (default=0.)) – L1 regularization term on weights.  reg_lambda (float, optional (default=0.)) – L2 regularization term on weights  I have seen data scientists using both of these parameters at the same time, ideally either you use L1 or L2 not both together.   While reading about tuning LGBM parameters I cam across one such case: Kaggle official GBDT Specification and Optimization Workshop in Paris where Instructors are ML experts. And these experts have used positive values of both L1 &amp; L2 params in LGBM model. Link below (Ctrl+F 'search_spaces' to directly reach parameter grid in this long kernel)      I have seen same in XGBoost implementations.  My question is why use both at the same time in LGBM/XGBoost.  Thanks.
113907
According to the references ,  and .  It has been mentioned that there is an equivalence between the regularized regression (Ridge, LASSO and Elastic Net) and their constraint formulas.  I have also looked at , and , but I can not see a clear answer show that equivalence or logic.  My question is   How to show that equivalence using Karush–Kuhn–Tucker (KKT)?  The following formulas are for Ridge regression.    NOTE  This question is not homework. It is only to increase my comprehension of this topic.   UPDATE  I haven't got the idea yet.
63923
In a typical neural network, which way is the common way to add regularization?   Assuming regression task, regression error loss is Mean-squared-error   Then we can have two choice of regularization on weights:   $\lambda$ * $\sum ||W||^2$ $\lambda$ * $\textbf{average} ||W||^2$    I have seen most people use the first option, just being curious to ask.
51049
I'm using linear regression to predict a price which is obviously positive. I have only one feature which is gross_area. I standardized it (z-score) I got this kind of value:  array([[ 1.        , -0.48311432],        [ 1.        ,  0.68052306],        [ 1.        ,  2.1426852 ],        [ 1.        , -1.17398593],        [ 1.        , -0.16265712]])   Where the 1 is the constant for the intercept term. I predict the parameters(predictors) and I got this:  array([[ 31780004.85045217],        [ 27347542.4693376 ]])   Where the first cell is the intercept term and the second cell correspond to the parameter found for my feature gross_area.  My problem is the following, when I take for example the fourth line and I compute the matrix multiplication XB to get my prediction, I got this:  In [797]: np.dot(training[4], theta) Out[797]: array([-325625.35640697])   Which is totally wrong since I cannot have negative value for my dependent variable. It seems like because of my normalization where I got negative value for my feature, I ended up with a negative predicted value for some tuple. How can is it possible and how can I fix this ? Thank you.  This is what I have predict graphically:   with y=price , x =gross area