I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables.  I am trying to do this in Python and sklearn.  The categorical variables have many different values. I was thinking to create dummy variables for each value in all the categorical variables. However, in the end this results in a sparse data set with thousands of variables.  How would you go about to reduce the number of dummy variables? Would you use Chi2 to select useful features?  Or would you not make dummy variables at all?
1613
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?    Consider a variable like  (discipline chosen by an undergraduate student). It is unordered and categorical, but it can potentially have dozens of distinct levels. Let's say I want to use major as a predictor in a regression model.  Using these levels as-is for modeling leads to all sorts of issues because there are just so many. A lot of statistical precision would be thrown away to use them, and the results are hard to interpret. We're rarely interested in specific majors -- we're much more likely to be interested in broad categories (subgroups) of majors. But it isn't always clear how to divide up the levels into such higher-level categories, or even how many higher-level categories to use.  For typical data I would be happy to use factor analysis, matrix factorization, or a discrete latent modeling technique. But majors are mutually exclusive categories, so I'm hesitant to exploit their covariance for anything.  Furthermore I don't care about the major categories on their own. I care about producing higher-level categories that are coherent with respect to my regression outcome. In the binary outcome case, that suggests to me something like linear discriminant analysis (LDA) to generate higher-level categories that maximize discriminative performance. But LDA is a limited technique and that feels like dirty data dredging to me. Moreover any continuous solution will be hard to interpret.  Meanwhile something based on covariances, like multiple correspondence analysis (MCA), seems suspect to me in this case because of the inherent dependence among mutually exclusive dummy variables -- they're better suited for studying multiple categorical variables, rather than multiple categories of the same variable.  edit: to be clear, this is about collapsing categories (not selecting them), and the categories are predictors or independent variables. In hindsight, this problem seems like an appropriate time to "regularize 'em all and let God sort 'em out". Glad to see this question is interesting to so many people!
51707
In answer to  factor pooling in model selection was discussed.  If a factor or categorical variable is to be dropped in model selection, should all levels be dropped simultaneously? If so, why?  The motivation for dropping factors is to aid model interpretation. For example, I might be interested in explaining the factors that influence customer behaviour when visiting a store and have a categorical variable "travel mode" with factors "walking, bus, private car, taxi, etc." In this context, I can remove all the dummy variables except "private car" because they have an insubstantial estimated magnitude and are not significant predictors of behaviour. I then end up with a "travelled in private car" vs "didn't travel in private car" variable and don't have to worry about troubling the reader with interpreting the other largely uninteresting variables.
109084
I have learned that, for creating a regression model, we have to take care of categorical variables by converting them into dummy variables. As an example, if, in our data set, there is a variable like location:   Location  ---------- Californian NY Florida   We have to convert them like:  1  0  0 0  1  0 0  0  1   However, it was suggested that we have to discard one dummy variable, no matter how many dummy variables are there.   Why do we need to discard one dummy variable?
9017
I'm struggling with the interpretation of a regression model where a categorial variable (5 levels) is dummy coded. Here is the result of my calculation in R:  Call: lm(formula = DV ~ Age + Gender + factor(Categorial) +  Continuous 1 + Continuous 2 + Continuous 3,  data = dat)  Residuals:  Min       1Q   Median       3Q      Max  -1.30058 -0.25326  0.00349  0.28123  1.49877   Coefficients:                       Estimate Std. Error t value Pr(&gt;|t|)    (Intercept)           -0.42367    0.30694  -1.380  0.16842    Age                   -0.05949    0.02026  -2.936  0.00356 ** Gender                -0.01800    0.04828  -0.373  0.70952    factor(Categorial)2   -0.30625    0.12645  -2.422  0.01596 *  factor(Categorial)3   -0.03441    0.07752  -0.444  0.65736    factor(Categorial)4   -0.12603    0.09914  -1.271  0.20453    factor(Categorial)5   -0.08417    0.13269  -0.634  0.52630     Continuous 1           0.12080    0.04346   2.779  0.00575 ** Continuous 2          -0.06592    0.04383  -1.504  0.13354    Continuous 3          -0.06230    0.03475  -1.793  0.07392 .  --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Residual standard error: 0.4259 on 336 degrees of freedom   (6 observations deleted due to missingness) Multiple R-squared:  0.1315,    Adjusted R-squared:  0.1057  F-statistic: 5.089 on 10 and 336 DF,  p-value: 6.353e-07   Ok. Age, Factor 2 of the categorial variable and the first continuous variable are significant predictors of the dependent variable. so far so good.   What I'm not understanding is:   The reference category of the dummy coded categorial variable is the intercept and the first category of the categorial variable. right? How do I interpret this?  When doing an anova with the categorial variable as a independent variable, this factor is a significant predictor. With the results of the linear model, one could conclude that this is only due to category 2, right? Can I test contrasts with this linear regression model (e.g. Category1 vs. Category2)? Should I include interactions?   I'd be glad for any help :-)
112560
I am currently using principal components analysis to select variables to use in modelling.   At the moment, I make measurements A, B and C in my experiments -- What I really want to know is: Can I make fewer measurements and stop recording C and or B to save time and effort?    I find that all 3 variables load heavily onto my first principal component which accounts for 60% of the variance in my data. The component scores tell me that if I add these variables together in a certain ratio (aA+bB+cC). I can get a score on PC1 for each case in my dataset and could use this score as a variable in modelling, but that doesn't allow me to stop measuring B and C.    If I square the loadings of A and B and C on PC1, I find that variable A accounts for 65% of the variance in PC1 and variable B accounts for 50% of the the variance in PC1 and variable C also 50%, i.e. some of the variance in PC1 accounted for by each variable A, B and C is shared with another variable, but A comes out on top accounting for slightly more.     Is it wrong to think that I could just choose variable A or possibly (aA+bB, if necessary) to use in modelling because this variable describes a large proportion of the variance in PC1 and this in turn describes a large proportion of the variance in the data?   Which approach have you gone for in the past?   Single variable which loads heaviest on PC1 even if there are other heavy loaders?     Component score on PC1 using all variables even if they are all heavy loaders?
6491
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.   My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).   But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?
46062
In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables containing similar information, will the model weight too much on this set rather than the others?   For example, there are two sets of information (A,B) with the same predictive power. Variable $X_1$,$X_2$,...$X_{1000}$ all contain information A, and only Y contains information B. When random sampling variables, will most of the trees grow on information A, and as a result information B is not fully captured?
107838
In Decision Trees, when doing classification or regression, do we use only numerical values? Suppose I have a categorical column Wind as a feature. Suppose I am have 5 rows (observations), and the values for Wind are [high,low,high,medium,medium].  Can I feed this categorical data into a decision tree classifier (like scikit-learn), will it take the data? That is, will the splitting of node based on wind happen on categorical values high with count 2, medium with count 2 and low with count 1. Or, will the system converts it into dummy variable?
79580
I am performing simple (first-order terms) linear regression on data having several categorical variables (i.e. factors), and it is often desired that for each factor, one of the levels should add nothing to the regressand, and the other levels should add positive values to the regressand.  When I perform a regression analyses, however, I often get a lot of negative coefficients.  Is there a non-manual way of choosing which levels of a factor should be used as regressor variables in order to maximize the number of positive coefficients in the equation?  In other words, how can I get R to do this (somewhat tedious) task for me?
68082
I had a dataset that consists of 20000 chess matches. In this dataset there are columns and variables- variables like the players’ ratings- that might have impact on a binary result, white player winning more. To test this, my lecturer told Me to perform a nested regression model where you could see the affect of each of these variables. Since it’s a binary outcome, I thought about making a nested logistic regression model. However, I don’t know how to do that and how to do that in python. Can someone please help?
21385
I have a linear regression with two explanatory variables. One of them is a two level categorical variable. When I perform the fit, R tells me that the intercept is highly non significant. When I force the intercept to be zero in the regression, it tells me that the coefficient for the 0-level of the categorical variable is highly non significant. What should I do?
2778
I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination of categorical and numerical columns.  dummified_data = data.get_dummies() train_data = dummified_data[:10000] test_data = dummified_data[10000:12000]   Currently, I have a hand-written function that takes user-inputted data and transforms it into a format like dummy data. This doesn't seem sustainable as the number of columns/the size of my categorical variables grows.  Is there a way to dummify training data and production data consistently?
122937
I need to recode the variable school setting (urban, sub-urban and rural settings) into a dummy variable. I know that when creating a dummy variable, there is one category less (so 2 rather than three conditions) and that the urban is the largest group and should be used as baseline (not sure if baseline is the right word, but hopefully you know what I mean). However, I don't know what to do from here.  Is the variable changed into: 1. Two dummy variables: setting 1 with urban/sub-urban and setting 2 with urban/rural? 2. One variable where urban is given the value 1, and urban and sub-urban are given the value 0? 3. Both? 4. Have I completely missed something?  Hope you can help me!!
64899
Let's say that I have one categorical variable with six levels, and I then create five indicator variables in order to represent the six levels. If two of the five variables are insignificant, then do I drop these two? I assume not, but I was not sure. I was thinking that it might be better to test the full (all five variables) versus the reduced (just the three significant variables)  model and, if that was not significant, then just leave all five of the variables in. I was not sure what to do. Oh and I meant for this to be in the context of fitting a logistic regression model.
97104
I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) Label  Feat1 Feat2 Feat3 Feat4  1      1     1      2     2  2      3     1      2     2   3      4     3      3     1       4      4     1      2     4  I'm using scikit learn, and am considering using Naive Bayes or a Decision Tree. The classifier needs to be able to deal with missing features, and I read on scikit learn's  that Decision Tree does not support missing values. What I am looking for is advice as to how to approach missing categorical values when using scikit learn. Also, any links to academic papers addressing this would be appreciated.
69881
I'm trying to play around with classification models and started off with logistic regression in R. When I have all the numeric variables in the data set the model works correctly and I was able to interprer the results.  The question is, with another data set I have more than 5 variables which are categorical but they are important though.   1) How do I deal with the variables with categorical values? the total unique categorical values are more than 100 for almost 5 predictors.  2) My dependent variable is having 3 classes, is it okay to using logistic regression for this purpose? (I would still need to explore other classification techniques but right now I'm exploring logistic regression.
104728
I'm a software engineer working on machine learning. From my understanding,  (such as OLS) and   (such as logistic regression and SVM) make a prediction based on an inner product between trained coefficients $\vec{w}$  and feature variables $\vec{x}$:  $$ \hat{y} = f(\vec{w} \cdot \vec{x}) = f(\sum_{i} w_i x_i)  $$  My question is: After the model has been trained (that is, after the coefficients $w_i$ have be computed), is it the case that the coefficients will be larger for feature variables that are more important for the model to predict more accurately?  In other words, I am asking whether the relative magnitudes of the coefficients can be used for  by just ordering the variables by coefficient value and then selecting the features with the highest coefficients? If this approach is valid, then why is it not mentioned for feature selection (along with wrapper and filter methods, etc.).  The reason I ask this is because I came across a discussion on . There is a blurb that says:     Built-in feature selection is frequently mentioned as a useful   property of the L1-norm, which the L2-norm does not. This is actually   a result of the L1-norm, which tends to produces sparse coefficients   (explained below). Suppose the model have 100 coefficients but only 10   of them have non-zero coefficients, this is effectively saying that   "the other 90 predictors are useless in predicting the target values".   Reading between the lines, I would guess that if a coefficient is close to 0, then the feature variable with that coefficient must have little predictive power.  EDIT: I am also applying z-scaling to my numeric variables.