I know that in a linear regression if independent variable is categorical variable with k levels then we will be needing k-1 dummy variables, what happens if we model the relationship with k dummy variables
122085
There are two different ways to encoding categorical variables. Say, one categorical variable has n values.  converts it into n variables, while  converts it into n-1 variables. If we have k categorical variables, each of which has n values. One hot encoding ends up with kn variables, while dummy encoding ends up with kn-k variables.  I hear that for one-hot encoding, intercept can lead to collinearity problem, which makes the model not sound. Someone call it "".  My questions:   Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website. Since one-hot encoding generates more variables, does it have more degree of freedom than dummy encoding?
13360
I create a linear regression model with interaction term in the model say: $$y=a_0+a_1x_1+a_2x_2+a_3x_1*x_2+e$$ where $x_1$ is continuous and $x_2$ is binary. Now, I have couple of questions:   If $x_1$ and $x_2$ both are significant and have positive coefficients, that means that if $x_1$ and $x_2$ increase, then $y$ increases. What can I say if $x_1*x_2$ is statistically insignificant? Does it mean that increase in. Does it mean that for $x_2=1$, the effect of $x_1$ on $y$ is not changed, i.e. increase in $x_1$ still leads to increase in $y$? What if the interaction is statistically significant? What can I say if the interaction term is statistically insignificant and negative? Does it indicate that if $x_2=1$, increase in $x_1$ actually leads to decrease in $y$?   Thanks.
43873
I am using a logit model. My dependent variable is binary. However I have an independent variable which is categorical and contains the responses: 1.very good, 2.good, 3.average, 4.poor and 5.very poor. So, it is ordinal ("quantitative categorical"). I am not sure how to handle this in the model. I am using gretl.  [Note from @ttnphns: Although the question says the model is logit (because the dependent is categorical), the crucial issue - ordinal independent variables - is basically alike, be the dependent categorical or quantitative. Therefore the question is equally relevant to, say, linear regression too - as it is to logistic regression or other logit model.]
1109
I have a data having 2 continuous and 4 categorical variables. Each categorical variable has 3 levels. I want to know how to include the variables  in the model. I am using SPSS  Variables:   Sales - Dependent Variable Retail Price Location : TX, CA, PY Display Location in Store : DL1, DL2, DL3 Price Point : 2.5, 4.5, 1.0   I have created 9 dummy variables. 3 for Location, 3 for Display Location in Store and 3 for Price Point.  I am confused on how to use these independent variables. How to add the independent variables into the models. Should I add all the nine dummy variables or two dummy variables of each categorical variables along with Retail Price.  I searched on net but , did not get any solution. I would appreciate any kind of help.  Thanks in advance.
108619
It looks like you can use coding for one categorical variable, but I have two categorical and one continuous predictor variable. Can i use multiple regression for this in SPSS and if so how? thanks!
17858
A few days ago, a psychologist-researcher of mine told me about his method to select variables to linear regression model. I guess it's not good, but I need to ask someone else to make sure. The method is:      Look at correlation matrix between all variables (including Dependent   Variable Y) and choose those predictors Xs, that correlate most with Y.     He didn't mention any criterion.  Q: Was he right?   [I think that this selection method is wrong, because of many things, like it's the theory that says which predictors should be selected, or even  omitted variable bias (OVB).]
29366
I have a study where many outcomes are represented like percentages and I'm using multiple linear regressions to asses the effect of some categorical variables on these outcomes.  I was wondering, since a linear regression assume that the outcome is a continuous distribution, are there methodological problems in applying such model to percentages, which are limited between 0 and 100?
80561
How can I account for a nonlinear independent variable in a logistic regression?  For example, consider this data set:  b1  b2  b3  b4  b5 1   1   0   0   20 0   1   0   0   20 1   0   0   0   13 1   1   1   0   8 1   1   0   1   5 0   1   0   1   4 1   1   0   1   5 0   0   0   0   8 0   0   1   0   13 1   0   0   1   20 1   0   0   0   29 1   1   0   1   40 1   0   0   0   53 0   1   1   0   68   Suppose I gathered some data like the above (suppose I had 10,000 rows) and I want to use them to predict future sets of similar data with an equation generated by a logistic regression. I get the above data. My independent variables are b1 through b5. By looking at the data, I can clearly see that independent variables b1 through b4 are binary (maybe they are male/female, rent/own, etc).   However I see that b5 is clearly not binary. Upon graphing it and looking at the numbers I determine that the b5 variable has a bathtub shaped U curve with an equation of b5=(x-4)^2+4.   I understand that using the b5 variable as it is would screw up my logistic regression - since any interval independent variables must have a linear relationship with the dependent variable.   Yet, I must account for the variable b5 in my data set while doing a logistic regression for it to be accurate and predict well (right?).  Given the above data, how would I go about making a predictive model with a logistic regression? Any references online or in books would greatly be appreciated as I am having trouble finding a good explanation of the above.  Cheers =)  EDIT:  Sorry if my question is unclear.  Consider this: Suppose we had the following data on many people:  Y: Whether they had a stroke before the age of 50 or not 1. Whether they own a pet or not. 2. Whether they graduated college or not. 3. Whether they are male or female. 4. The number of hours they exercise every week.  We want to train a logistic regression model based on historical data to predict if someone in the future with the same data will be more apt or not to have a stroke before the age of 50.  The problem I am having is with the last variable. A continuous ratio variable like "number of hours exercised each week."  Exercising more each week may have diminishing returns on health and thus having a stroke. For example decreasing an increasing rate. Or maybe (totally made up) exercising a little bit is really good for you, exercising a moderate amount is really bad for you, and exercising a lot is really good for you again. We might have a U shaped curve like (x-4)^2+4 for example when plotting health to exercise.   I would image that having a curved, non-linear independent variable in a logistic regression like the one I described above would cause problems. Am I right? And if so, what kinds of things can you do to still include the variable in the logistic regression. Perhaps a transformation? Is that all?
40756
I found a reference in an article that goes like:     According to Tabachnick &amp; Fidell (1996) the independent variables with   a bivariate correlation more than .70 should not be included in   multiple regression analysis.   Problem: I used in a multiple regression design 3 variables correlated >.80, VIF's at about .2 - .3, Tolerance ~ 4- 5. I cannot exclude any of them (important predictors and outcome). When I regressed the outcome on the 2 predictors which correlated at .80, they remained both significant, each predicted important variances, and these same two variables have   the largest part and semipartial correlation coefficients amongst all 10 variables included (5 controls).   Question: Is my model valid despite high correlations? Any references greatly welcomed!    Thank you for the answers!  I did not use Tabachnick and Fidell as a guideline, I found this reference in an article dealing with high collinearity amongst predictors.   So, basically, I have too few cases for the number of predictors in the model (many categorical, dummy coded control variables- age, tenure, gender, etc) - 13 variables for 72 cases. The Condition Index is ~ 29 with all the controls in and ~23 without them (5 variables).   I cannot drop any variable or use factorial analysis to combine them because theoretically they have sense on their own. It is too late to get more data. Since I am conducting the analysis in SPSS perhaps it would be best to find a syntax for ridge regression (although I haven't done this before and interpreting the results would be new to me).   If it matters, when I conducted stepwise regression, the same 2 highly correlated variables remained the single significant predictors of the outcome.  And I still do not understand if the partial correlations which are high for each of these variables matter as an explanation for why I have kept them in the model (in case ridge regression can't be performed).   Would you say the "Regression diagnostic: identifying influential data and sources of collinearity / David A. Belsley, Edwin Kuh and Roy E. Welsch, 1980" would be helpful in understanding multicollinearity? Or might other references be useful?
10191
I am currently investigating the impact of certain indicators such as GDP and inflation on the stock market. However some of my variables are non-stationary and some stationary in levels. All variables are stationary in first differences.  My question:   Since I have a mix of I(0) and I(1), I need to take the first difference of the I(1) variables and then use VAR. Say that I have a dependent variable $y$ which is I(0) and independent variables $x_1$ which is I(1) and $x_2$ which is I(0). Do I take the first differences of only $x_1$ and then apply the VAR model? Is that correct?  If I take the first differences, then might I lose the long-run relationship between the variables? Is ARDL the better method in this case?   If you have any references regarding the above I would be grateful.
21385
I have a linear regression with two explanatory variables. One of them is a two level categorical variable. When I perform the fit, R tells me that the intercept is highly non significant. When I force the intercept to be zero in the regression, it tells me that the coefficient for the 0-level of the categorical variable is highly non significant. What should I do?
35328
I understand that when label encoding is used ,the numeric number can be interpreted to have an order and a model could assume a linear relationship. However shouldn't this be a problem when there are in-fact many levels in a categorical variable e.g. country.  How about the case of binary variables which for e.g. instead of gender.male ( 1,0) what if I just used Gender ( 1,0) where gender = 0 is female and 1 is male for instance. This shouldn't impact the model as much as a label encoding to a feature with multiple levels? How would this be for a case where I have a feature three levels ( -1,0,1) where -1 means not applicable, 0 means 'No' and 1 means 'Yes', so instead of having 2 columns feature.not_applicable(1,0) and feature.No(1,0) - mathematically how would models be impacted - models here would be GLMS,  boosting models, Random forests etc. Is label encoding recommended when feature has say &lt;=3 to 4 levels and above that is recommended to do one hot encoding or $n-1$ dummy variables?
19415
I have a output variable and 1 categorical predictor and 3 continuous predictors.  dat=data.frame(IV = rnorm(9), group=c("A","A","A","B","B","B","C","C","C"),pred1=rnorm(9),pred2=rnorm(9),pred3=rnorm(9))   What is the recommended way to add the dummy variable and how does the interpretation of the output change with these 2 setups?  If I do this and include a single column for the dummy variable     lm(IV~factor(group)+pred1+pred2+pred3, data = dat)  lm(formula = IV ~ factor(group) + pred1 + pred2 + pred3, data = dat)  Coefficients:    (Intercept)  factor(group)B  factor(group)C           pred1           pred2           pred3          -0.3143          0.9496         -0.3502          0.1839         -0.5349         -0.4710     In the case above the results the groupB and groupC coefficients are RELATIVE to groupA.  OR if I don't want the results to be relative I can include groupA like this:  dat2 = cbind(dat[,c("IV","pred1","pred2","pred3")], model.matrix(~ factor(dat$group) - 1) ) lm(IV~., data = dat2)  Call: lm(formula = IV ~ ., data = dat2)  Coefficients:          (Intercept)                 pred1                 pred2                 pred3  `factor(dat$group)A`  `factor(dat$group)B`  `factor(dat$group)C`                -0.6645                0.1839               -0.5349               -0.4710                0.3502                1.2998                    NA     I have a few questions:   How does the interpretation of the intercept change between the 2 models? Why in the last model is the GROUP C coefficient "NA"?
110127
The task is to build a regression model for individuals. I have all the independent variables for each individual, but the dependent variable only as an aggregates on group-level.  Lets say, I am trying to predict the score a student will achieve at some test. I have information about the student that can be used as predictor-variable (f.e. time spent studying). But the results of the test are only given as aggregated sums for each class. I can link every student to a class, but I don't know individual test-results.  One potential way I can think of would be to aggregate the independent variables too and run the regression completely on the aggregated data.  But it's probably rarely the case that correlation on aggregated level and individual level are the same. So I don't know how to make any judgement about the validity of such an approach.  Is there any 'good'(or less bad) approach to this problem?
3015
I have a dataset of around 120000 (120K) unique individuals. I am fitting a binary logistic regression, where I have around 150 variables to choose from. For the categorical variables, some are very imbalanced. For example, there is a variable that can take on Yes (=1) or No (=0). There are 119K individuals with Yes and the remaining 1K with No.    What are the consequences of having such a variable in the logistic regression?  Can it make my results unstable if I have many of these (say 10-20 similar variables), when using stepwise (forward) or backward automatic selection procedures?  What if the predictor / variable is not binary, but has for example 5 categories where 1 category is largely over or underrepresented?
116069
In Element of Statistical Learning, chapter 4-2 about linear regression of an indicator matrix, it is stated that the sum of predictor is equal to 1.  To bring a bit of context: We have $N$ training samples of size $p$ represented by a $N$ x $(p+1)$ matrix (intercept included) $\textbf{X}$. Each training sample $x_i$ belong to a category $g_i \in [| 1, .., K |]$. We use the $N$ x $K$ indicator matrix for the training samples which contains only one 1 per row. We try to fit a linear regression model for the parameter $\textbf{B}$ and we get $\hat{\textbf{B}} = (\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T\textbf{Y}$.  For a new data, we compute the fitted output $\hat{f}(x)^T = (1,x^T)\hat{\textbf{B}}$ and the category is then $arg \max_{k \in [| 1, .., K |]} \hat{f}_k(x)$  They say that $\sum_{k \in [| 1, .., K |]} \hat{f}_k(x) = 1$ for any $x$  as long as there is an intercept in the model.  My question is thus why? I try to use the coefficient relation from the regression derivation ($\textbf{X}^T(\textbf{Y} - \textbf{X}\hat{\textbf{B}}) = 0$) but without success.
97783
I'm trying to get caught up in some notes, and the topic is pooled, cross-sectional data. If we have the regression: $$\log{wage_i}=\beta_0+\beta_1\text{married}_i+\delta_0\text{yr10}_i+\delta_1\text{married}_i\times \text{yr10}_i+\varepsilon_i$$ Where $\text{married}$ is a dummy variable which $=1$ if a person is married and $0$ if not, and $\text{yr10}$ is a dummy variable indicating whether the observation was recorded in the year 2010, as opposed to being recorded in the year 2000. There's a blank spot in the notes that I didn't get, and it asks: How do we interpret $\delta_1$?  I think it's correct to say that $\delta_1$ measures how the return on whether or not someones married has changed in the past $10$ years. Is that correct? Or am I wrong? I read a little bit in my textbook and I think that's what it's saying.