How to deal with regression when most of the independent variables are categorical having numerous (more than 10) levels and the dependent variable is continuous? For this would it make sense to perform dummy coding? What could be a better way to deal with this situation? I want to predict CPM (Cost per mille) based on certain categorical independent variables.
1613
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?    Consider a variable like  (discipline chosen by an undergraduate student). It is unordered and categorical, but it can potentially have dozens of distinct levels. Let's say I want to use major as a predictor in a regression model.  Using these levels as-is for modeling leads to all sorts of issues because there are just so many. A lot of statistical precision would be thrown away to use them, and the results are hard to interpret. We're rarely interested in specific majors -- we're much more likely to be interested in broad categories (subgroups) of majors. But it isn't always clear how to divide up the levels into such higher-level categories, or even how many higher-level categories to use.  For typical data I would be happy to use factor analysis, matrix factorization, or a discrete latent modeling technique. But majors are mutually exclusive categories, so I'm hesitant to exploit their covariance for anything.  Furthermore I don't care about the major categories on their own. I care about producing higher-level categories that are coherent with respect to my regression outcome. In the binary outcome case, that suggests to me something like linear discriminant analysis (LDA) to generate higher-level categories that maximize discriminative performance. But LDA is a limited technique and that feels like dirty data dredging to me. Moreover any continuous solution will be hard to interpret.  Meanwhile something based on covariances, like multiple correspondence analysis (MCA), seems suspect to me in this case because of the inherent dependence among mutually exclusive dummy variables -- they're better suited for studying multiple categorical variables, rather than multiple categories of the same variable.  edit: to be clear, this is about collapsing categories (not selecting them), and the categories are predictors or independent variables. In hindsight, this problem seems like an appropriate time to "regularize 'em all and let God sort 'em out". Glad to see this question is interesting to so many people!
43698
Is it appropriate to do a logistic regression where both the dependent and independent variables are binary? for example the dependent variable is 0 and 1 and the predictors are contrast coded variables -1 and 1 ?
86217
In linear regression, the transformations of explanatory variables is done to have maximum correlation with the dependent variable.  What is the best measure of choosing between multiple transformations in logistic regression as dependent variable is binary and not continuous?  The end goal is to maximize the lift (predictive power) of the model.
19075
I am trying to do an ordinal logistic regression (ordinal outcome variables with more than 2 categories) with nominal (more than 2 categories for some) predictor variables, as well as nominal (more than 2 categories for some) covariates/moderators.  First of all, I would like to test for effect interaction to see which of my covariates/moderators are indeed moderators so that I can treat them as such.  Second of all, I would like to know what's the best way to come up with the optimal model? Do I need to do dummy variables for the covariates? Can I do a stepwise for ordinal regression?
64481
Some predictive modeling techniques are more designed for handling continuous predictors, while others are better for handling categorical or discrete variables.  Of course there exist techniques to transform one type to another (discretization, dummy variables, etc.).  However, are there any predictive modeling techniques that designed to handle both types of input at the same time without simply transforming the type of the features?  If so, do these modeling techniques tend to work better on data for which they are a more natural fit?  The closest thing that I know of would be that usually decision trees handle discrete data well and they handle continuous data without requiring an up front discretization.  However, this isn't quite what I was looking for since effectively the splits on continuous features are just a form of dynamic discretization.  For reference, here are some related, non-duplicate questions:
12268
In the context of feature selection it is common to recode categorical variables with more than 2 categories into dummies. Selection methods such as elastic nets or lasso regression select the best predictors, whereby it is possible that only some dummies of each categorical variable are selected. I am wondering, if there can appear some problems due to this procedure. I found some comments about the topic on  and a , stating that the procedure should be used carefully, but that there are no general problems. However, I was not able to find any detailed literature or any educated guidelines, which could be followed.  Question: Can there appear any problems, if not all dummies of a categorical variable are selected for a model?  For example, I could imagine that the automatic selection relies on the order of the categories and the resulting reference category. Let's say there is a variable with categories A, B, and C. A dummy recoding into dummyB and dummyC would probably result in different variable selections compared to a dummy recoding into dummyA and dummyB.  Any advice or literature is highly appreciated!  UPDATE:  Based on Ben's comment I found some literature about a comparison of the lasso and the group lasso, which addresses my question:      However, based on this literature 2 further questions appeared:  1) It seems like the normal lasso is still used regularly, whereby the group lasso doesn't appear that often in current literature. Is there a specific reason for that?  2) When I have categorical variables with many categories, isn't it a problem, if I select the whole categorical variable? Or in other words, is it sometimes advantageous to use the lasso instead of the group lasso?
1109
I have a data having 2 continuous and 4 categorical variables. Each categorical variable has 3 levels. I want to know how to include the variables  in the model. I am using SPSS  Variables:   Sales - Dependent Variable Retail Price Location : TX, CA, PY Display Location in Store : DL1, DL2, DL3 Price Point : 2.5, 4.5, 1.0   I have created 9 dummy variables. 3 for Location, 3 for Display Location in Store and 3 for Price Point.  I am confused on how to use these independent variables. How to add the independent variables into the models. Should I add all the nine dummy variables or two dummy variables of each categorical variables along with Retail Price.  I searched on net but , did not get any solution. I would appreciate any kind of help.  Thanks in advance.
121148
Some of the variables are questions that are not being asked to every single person in the questionaire depending on how they answered previous question(s). So would I still be able to use a logistic regression if some of the questions (the independent variables) are only measuring the responses of SOME of the respondents?
4148
I have a continuous outcome variable. I understand that if I have a binary predictor, and a continuous predictor, and an interaction, then the model looks like this:  $y_{i} = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{1}x_{2} + \varepsilon_{i}$  However, I'm thinking of making the binary predictor a categorical one instead, with three categories.    What will the model equation look like once I make that change? I understand that I'll need to have two dummy variables, but I can't conceptualize what the interaction will look like under that circumstance.  Does it make any difference whether I use 0,1 to code the binary predictor, or some other values like 1,2? This question also applies to the model with the categorical predictor, since in that case I'll have to decide how to code the dummy variables.  Prior to the interaction term being in my model I was encouraged to center the continuous predictor. Is centering (still) a good idea now that I have this interaction term?
112189
I' m asking this question out of sheer curiosity, my teacher was not able to explain it.   If I'm using logistic regression with categorical variables they are coded like {1,2,3}. I guess it wouldn't change my results if I used {4,5,6}. But what if linearity of coding wasn't kept? (say {4,10,99})? What I deal with is just the way of coding factor variables but is it possible that the statistical inference could be "distorted" this way? Or, worst case scenario, would I be able to draw nonsense conclusions from categorical data just because they are coded in some way?
26990
Say we have a binary classification problem with mostly categorical features. We use some non-linear model (e.g. XGBoost or Random Forests) to learn it.    Should one still be concerned about multi-collinearity? Why? If the answer to the above is true, how should one fight it considering that one is using these types of non-linear models?
119859
I have a model similar to below with a continuous outcome, continuous predictor and a categorical variable. the goal of this analysis is to make inferences on the group variable. The issue is that y is bounded by x in the form of y >= -x. When the model makes predictions you can see in the plot below that the predicted value is outside the possible range. See the last plot.   How can this be remedied? A custom loss function to penalize going beyond the bound and use gradient descent regression? Or is there some other way? Thank you.   x =   rep(seq(-10,10,1),each=5)  y =  rep(0,length(x) ) weights = sample( seq(1,1000,1) ,length(x)) groups = rep( letters[1:5], times =length(x)/5  )   ## the outcome y has the relationship y &gt;= -x for(i in 1:length(x)){   y[i] = sample(  seq(-x[i],-x[i]+ ifelse(x[i]&lt;0 ,                                            sample(seq(1,10,.1),1),                                           sample(seq(5,40,.1),1) ),1)                        ,1) }   dat =data.frame(x = x, y = y, weights = weights, group = groups) model = lm(dat = dat, y ~ x + group + group * x, weights = weights) dat$FIT= predict(model, dat, allow.new.levels = TRUE)  library(ggplot2) ggplot(data = dat, aes(x = x, y = y, color = group))+geom_point( aes(size = weights))+ geom_line(data = dat, aes(x = x, y = FIT, color = group))    #data for new predictions newdata = data.frame( x = seq(-100,100,1) ,                        group = rep("c", length( seq(-100,100,1) )),                       bound = -seq(-100,100,1)                       )  ## when pridicting new data the outcome y has the relationship y &gt;= -x ## but the predicted vales are below the bound. See how red line is below blue to the left of zero new_preds  = predict(model, newdata, allow.new.levels = TRUE) ggplot(data = newdata, aes(x = x, y = new_preds, color = group))+geom_line()  +   geom_line(data = newdata, aes(x = x, y = bound, color = "green"))
79580
I am performing simple (first-order terms) linear regression on data having several categorical variables (i.e. factors), and it is often desired that for each factor, one of the levels should add nothing to the regressand, and the other levels should add positive values to the regressand.  When I perform a regression analyses, however, I often get a lot of negative coefficients.  Is there a non-manual way of choosing which levels of a factor should be used as regressor variables in order to maximize the number of positive coefficients in the equation?  In other words, how can I get R to do this (somewhat tedious) task for me?
22442
I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables.  I am trying to do this in Python and sklearn.  The categorical variables have many different values. I was thinking to create dummy variables for each value in all the categorical variables. However, in the end this results in a sparse data set with thousands of variables.  How would you go about to reduce the number of dummy variables? Would you use Chi2 to select useful features?  Or would you not make dummy variables at all?
84113
I am trying to find the variable importance on a credit scoring database. I have categorical inputs as well as numerical inputs. My question is does the random forest algorithm works the same way when I   case (1) : give only categorical variables as inputs  case (2): both categorical and numerical variables as inputs.  By this what I am actually asking is only categorical variables are enough or numerical variables should be included along with categorical variables as inputs to the algorithm...???  Thanks in advance...
40357
I'm doing a multivariate logistic regression where all my independent variables are categorical and binary. I have transformed all my categorical variables into dummies in order to have  reference groups and interpret my odds-ratios. However, I would like to check if there are eventually  multicollinearity issues.   I plan to calculate Spearman correlation between my independent variables and calculate VIF too. Nevertheless, I have several questions: Is Vif appropriate in this case? Do I have to calculate VIF and Spearman correlation on my categorical variables or their associated dummies?   I have tested other ways to detect multicollinearity (I check if the coefficients vary a lot if I increase my sample size or drop or add variables) and I'm pretty sure there is not a collinearity issue but I would like to have a "quantitative" proof.
64334
I am performing ordinal regression, I have 5 response categories and several predictors both continuous and categorical. I would like to add a predictor which is categorical but ordered (1, 2, 3, 4). I don't think it would be appropriate to apply the usual dummy coding for unordered categorical predictors, but when I searched for how to code this I did not find much information.  In  "linear coding" or "assuming linearity of the predictor effect" is mentioned, but without further details.  Does it mean I just use my ordered values as they are, i.e. use them as a continuous variable?