I am trying to understand what neural nets have in common with multiple linear regressions. Is is correct that every node in every layer (not counting the input layer) is just a multiple regression with the variables being the nodes in the preceding layer? I've included a  to show what I mean. How is this understanding wrong/right?
109832
I want to know the difference between linear regression in a regular machine learning analysis and linear regression in "deep learning" setting. What algorithms are used for linear regression in deep learning setting.
75231
Suppose we have data matrix $X$ which is $n\times p$ matrix ($n$ data points, and $p$ features including intercept), i.e., we already add the intercept term (append $1$ column) to $X$   For logistic regression (binary classification), the model parameters / regression coefficients $\beta$ is a length $p$ vector. For softmax regression (multi-class classification), the model parameters $W$ is $p \times m$ matrix, where $m$ is the number of classes.   Now, suppose we set $m=2$, then $W$ is a $p \times 2$ matrix. They are the same model, so the number of parameters CANNOT be doubled and we should be able to derive one from another.   So, my question is  What is the relationship between $\beta$ and $W$?
12577
In class we've seen that $a$ (the weights) must satisfy $$X^T (y-Xa) =0$$  Here $X$ is a $(n\times d)$ matrix (so we have $n$ samples in $\mathbb R^d$)  let's denote the residuals $r = y-Xa$. In our class notes, it is written that     The normal equations require the residuals to be orthogonal to each of the columns of $X$.   Why?     Therefore, the solution of the linear regression is a projection of $y$ onto the subspace spanned by $v_1 , \ldots , v_d$ (the columns of $X$)   Can you please explain this?
61779
Please notice that this , but it seems that it doesn't interest MO community. So, I have got a comment to post in this community in the hope that I may get some attention to this question.  Background:  For Kernel ridge regression, I have normally come across the data-set given in vector and scalar form, i.e., $\overline{S}:= \{x_i, \overline{y}_i \}$, where $x_i \in M_{n,1}(\mathbb{R})$ and $\overline{y}_i \in \mathbb{R}$. Then, for example, I could do the ridge-regression as given here .    My question:   Can we do the same Kernel ridge regression but with a data set in matrix-vector form, i.e., $S := \{ X_i, y_i \}_{i=1}^{N}$, where $X_i \in M_{n,p}(\mathbb{R})$ and $y_i \in M_{n,1}(\mathbb{R})$? If yes, can you prove it or give me a reference/paper?  Thank you so much in advance for your help.
44714
Is there a relationship between regression and linear discriminant analysis (LDA)? What are their similarities and differences? Does it make any difference if there are two classes or more than two classes?
85263
In regression, you encode a categorical variable with n possible values using n-1 indicator variables. How about for neural networks?
16286
I haven't seen the question stated precisely in these terms, and this is why I make a new question.  What I am interested in knowing is not the definition of a neural network, but understanding the actual difference with a deep neural network.  For more context: I know what a neural network is and how backpropagation works. I know that a DNN must have multiple hidden layers. However, 10 years ago in class I learned that having several layers or one layer (not counting the input and output layers) was equivalent in terms of the functions a neural network is able to represent (see Cybenko's ), and that having more layers made it more complex to analyse without gain in performance. Obviously, that is not the case anymore.  I suppose, maybe wrongly, that the differences are in terms of training algorithm and properties rather than structure, and therefore I would really appreciate if the answer could underline the reasons that made the move to DNN possible (e.g. mathematical proof or randomly playing with networks?) and desirable (e.g. speed of convergence?)
113907
According to the references ,  and .  It has been mentioned that there is an equivalence between the regularized regression (Ridge, LASSO and Elastic Net) and their constraint formulas.  I have also looked at , and , but I can not see a clear answer show that equivalence or logic.  My question is   How to show that equivalence using Karush–Kuhn–Tucker (KKT)?  The following formulas are for Ridge regression.    NOTE  This question is not homework. It is only to increase my comprehension of this topic.   UPDATE  I haven't got the idea yet.
17051
I am trying to run a Multiple Regression model but I am not sure what should I do to prepare the data input. In my model the dependent variable is the artificial surfaces in hectares, the independent are the GDP (monetary value) and Population (inhabitants). The histogram shows that the data is not normal distributed.    I would like to know if I need to normalize the distribution of the data in order to get the "bell" curve and/or  if I need to normalize the data (with the max-min or standardize).
107168
I am a beginner to neural networks and have had trouble grasping two concepts:   How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever. How does one decide the number of neurons in each middle layer? Is it recommended having an equal number of neurons in each middle layer or does it vary with the application?
115366
For statistical and machine learning models, there are multiple levels of interpretability: 1) the algorithm as a whole, 2) parts of the algorithm in general 3) parts of the algorithm on particular inputs, and these three levels split into two parts each, one for training and one for function eval. The last two parts are much closer than to the first. I'm asking about #2, which usually leads to better understanding of #3). (if those are not what 'interpretability' means then what should I be thinking?)  As far as interpretability goes, logistic regression is one of the easiest to interpret. Why did this instance pass the threshold? Because that instance had this particular positive feature and it has a larger coefficient in the model. It's so obvious!  A neural network is  the classic example of a model that is difficult to interpret. What do all those coefficients mean? They all add up in such complicated crazy ways that it is hard to say what any particular coefficient is really doing.  But with all the deep neural nets coming out, it feels like things are becoming clearer. The DL models (for say vision) seem to capture things like edges or orientation in early layers, and in later layers it seems like some nodes are actually semantic (like the proverbial ). For example:    ()  This is a graphic () created by hand for presentation so I am very skeptical. But it is evidence that somebody thinks that is how it works.  Maybe in the past there just weren't enough layers for us to find recognizable features; the models were successful, just not easy to post-hoc analyze particular ones.  But maybe the graphic is just wishful thinking. Maybe NNs are truly inscrutable.  But the many graphics with their nodes labeled with pictures are also really compelling.    Do DL nodes really correspond to features?
85576
I am starting to study Neural Network. I want to build a MLP where I will feed it:   3 features which are continuous one feature which is categorical (48 classes)   How can I do this?  Before adding the categorical feature, I was using 'relu' activation function and 'lbfgs' as a solver in MLPRegrssor in sklearn.  I was wondering: how can I integrate this variable into the network? Should I do one-hot encoding? Should I keep relu as activation function or should I try sigmoid, softmax or something like that?
101080
hello I have a question about weight regularization in Adam apparently the weight_decay in the AdamW function  has the same impact as an L2 regularization my questions are: is that parameter the same as lambda that we have in the regularization term?  how does it exactly work? and what is its impact on the model complexity?
26956
I'm curious to know how important the bias node is for the effectiveness of modern neural networks. I can easily understand that it can be important in a shallow network with only a few input variables. However, modern neural nets such as in deep learning often have a large number of input variables to decide whether a certain neuron is triggered. Would simply removing them from, e.g., LeNet5 or ImageNet have any real impact at all?
79528
For a given input into the input nodes, there are multiple correct values for the output nodes. In the training set, there are times when the inputs result in a certain output, and other times when the inputs result in a completely different (but equally valid) output.  Will a neural network still be able to "figure out" a pattern? From what I know about backpropagation, it seems like the different right answers would prevent it from functioning properly. If that's so, are there any solutions?  I don't need the neural network to predict all possible correct solutions, but I do need it to output a correct solution.
59873
In a set of  that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this $$ y = \theta_1 + \theta_2\exp \left( {\theta_3x} \right) + \varepsilon $$ and suggests to first perform a nonlinear regression to estimate $\theta_3$, and then use OLS to estimate the other two parameters. I am not very experienced in regression modeling, so please forgive me if I ask a very simple question, but I don't understand how you can do this consistently without also estimating the other 2 parameters? Apparently, there is some trick involved, which I don't understand...  Another question is what happens to the ability to do inferences based on parameter estimates to the complete model if you do the regression in two stages like that?  Many thanks!
20438
What is Hat matrix and leverages in classical multiple regression? What are their roles? And Why do use them?   Please explain them or give satisfactory book/ article references to understand them.