Can I implement ridge regression in terms of OLS regression? Is it even possible?  I am interested because scikit-learn supports non-negative least squares (NNLS), but not non-negative ridge regression. So, I'd like to transform my data as to be able to call the underlying NNLS function, but achieve ridge regression functionality.
122571
How to perform non-negative ridge regression? Non-negative lasso is available in scikit-learn, but for ridge, I cannot enforce non-negativity of betas, and indeed, I am getting negative coefficients. Does anyone know why this is?  Also, can I implement ridge in terms of regular least squares? Moved this to another question:
47008
Why is using centered or uncentered data equivalent in ridge regression? In other words, given two ridge regression problems: \begin{equation} (b',c')=\operatorname*{argmin}_{b,c}\Big[ { \sum_i^{m} (y_i - c - b^Tx_i)^2 + \lambda b^Tb}\Big] \end{equation}  $$(b'',c'')=\operatorname*{argmin}_{b,c} \Big[{ \sum_i^{m} (y_i - c - b^T(x_i - \bar{x}))^2 + \lambda b^Tb} \Big]$$ where $\bar{x}$ is the mean of the input data, why does $(b',c')$ correspond to $(b'',c'')$?  I'm writing a piece of code where this thing holds numerically, I was wondering what is the mathematical explanation.
42973
I know how to perform a linear regression on a set of points. That is, I know how to fit a polynomial of my choice, to a given data set, (in the LSE sense). However, what I do not know, is how to force my solution to go through some particular points of my choosing. I have seen this be done before, but I cannot remember what the procedure was called, let alone how it was done.   As a very simple and concrete example, let us say that I have 100 points scattered on the x-y plane, and I choose to fit a whatever-order polynomial through them. I know how to perform this linear regression very well. However, let us say that I want to 'force' my solution, to go through, say, three of my data points at x-coordinates $x=3$, $x=19$, and $x=89$, (and their corresponding y-coordinates of course).  What is this general procedure called, how is it done, and are there any particular pitfalls I need to be aware of?  Edit:  I would like to add, that I am looking for a concrete way to do this. I have written a program that actually does the linear regression in one of two ways, inverting the covariance matrix directly, or through gradient descent. What I am asking is, how, exactly, step by step, do I modify what I did, such that I force the polynomial solution to go through specific points?   Thanks!
61155
Why are we using the squared residuals instead of the absolute residuals in OLS estimation?  My idea was that we use the square of the error values, so that residuals below the fitted line (which are then negative), would still have to be able to be added up to the positive errors. Otherwise, we could have an error of 0 simply because a huge positive error could cancel with a huge negative error.  So why do we square it, instead of just taking the absolute value? Is that because of the extra penalty for higher errors (instead of 2 being 2 times the error of 1, it is 4 times the error of 1 when we square it).
107838
In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am have 5 rows (observations), and the values for Wind are [high,low,high,medium,medium].  Can I feed this categorical data into a decision tree classifier (like scikit-learn), will it take the data? That is, will the splitting of node based on wind happen on categorical values high with count 2, medium with count 2 and low with count 1. Or, will the system converts it into dummy variable?
56710
Given a random walk $x_t$,   $$x_t=x_{t-1}+\varepsilon_t,$$  consider estimating the slope coefficient $\beta$ in   $$x_t=\beta x_{t-1}+\varepsilon_t$$  by OLS.  question and the following answer noted that $\hat \beta^{OLS}$ has a skewed distribution. My question is,  Are any of OLS assumptions violated in the model $x_t=\beta x_{t-1}+\varepsilon_t$ given that $x_t$ is a random walk? If so, what are the violations?
55834
I'm taking a course on regression models and one of the properties provided for linear regression is that the residuals always sum to zero when an intercept is included.   Can someone provide a good explanation for why this is the case?
40556
I have this confusion related to implementing linear regression with normalization. Let's say I have a training set trainX and trainY, and test set testX and testY. For the training set I take the mean and standard deviation of trainX, use it to transform the trainX data to have a  mean center and unit standard deviation. I do the same for trainY. Now I run a ridge regression for training. For cross validation, I use 10 fold and then get some coefficients optimal.  Now when I use these coefficients on the test sets testX and testY, I need to mean center and give unit standard deviation to both testX and testY using the mean and standard deviation I got from training data sets. I apply the coefficients to predict Y. To these predicted Y values, do I again need to add the mean and standard deviation used before to get the actual Y values? Is this the way to go?
20438
What is Hat matrix and leverages in classical multiple regression? What are their roles? And Why do use them?   Please explain them or give satisfactory book/ article references to understand them.
55306
Since RF can handle non-linearity but can't provide coefficients, would it be wise to use random forest to gather the most important features and then plug those features into a multiple linear regression model in order to obtain their coefficients?
49209
Any ideas on other polynomials I could successfully use for applying regression? My goal is a solution that has fit error strictly based on the noise. Is this possible since it is a bell-like curve? The tails can be long, short or non-existent. Am I looking for the impossible? Please note it has been a while since I have plunged this far into linear regression.   Anyway, my data points do not fit the $ax+bx^2+c$ polynomial well enough. I would like a replacement for this polynomial to apply regression (or different approach). In the following example, each point misses the fit curve by $20\%$ of the $Y$ range on average (of the data points). I looked at splines, but I do not know how to apply the regression since they look piece-wise.   Example $Y$ values (evenly spaced in $X$). It looks kind of like a bell curve. I would be applying 5 to 50 data points to determine the polynomial coefficients. Ultimately, I am interested in the best $X$ location for the peak based on the data points.     X      Y -40    -21142.1111111111 -30    -21330.1111111111 -20    -12036.1111111111 -10      7255.3888888889   0     32474.8888888889  10     32474.8888888889  20      9060.8888888889  30    -11628.1111111111  40    -15129.6111111111
66966
I usually hear about "ordinary least squares".  Is that the most widely used algorithm used for linear regression?  Are there reasons to use a different one?
117869
I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific subsets of the data in training itself, but I am stumped as how to perform regression on the data I have.  Normal OLS regression will produce outputs that will be over and under the [0,1] range if you do not change the likelihood to be the , or something else bounded in the same range, but will xgboost suffer the same mistake?  Any advice on trying to get this to work would be greatly appreciated by me.
109832
I want to know the difference between linear regression in a regular machine learning analysis and linear regression in "deep learning" setting. What algorithms are used for linear regression in deep learning setting.
41718
I did the multiple linear regression on a dataset of 412 observations, with one response variable (Y) and 25 explanatory variables(X1-X25). Y and most of Xs are not normally distributed. Besides, there are some correlation between several Xs. The plot show that the residuals strongly correlated with Y positively and weakly correlated with fitted Y negatively.(Sorry.As I'm newer in this website, I am n't allowed to post images.)  To address these problems, I have tried the principle component regression, the weighted least squared regression and the ridge regression. They all didn't work. I want to know what's wrong with the regression. Why did the residuals correlate with observed Y so obvious?
26990
Say we have a binary classification problem with mostly categorical features. We use some non-linear model (e.g. XGBoost or Random Forests) to learn it.    Should one still be concerned about multi-collinearity? Why? If the answer to the above is true, how should one fight it considering that one is using these types of non-linear models?
107264
When using support vector machine, are there any guidelines on choosing linear kernel vs. nonlinear kernel, like RBF? I once heard that non-linear kernel tends not to perform well once the number of features is large. Are there any references on this issue?