This is a question for those out there working in data scientist roles within your organizations. How many variables in acceptable to use within models that are going to be deployed in production for marketing or other purposes?  The reason I ask is this, we have four analysts, two of whom have been with the company for 10 years, and two of us who are recently graduated with our Masters in Predictive Analytics. Our senior analysts primarily build with linear / logistic regression models, and think that using the least amount of variables (regardless of technique) is always best, usually trying to use around 10-15 variables.  Us newer analysts work primarily with random forest and xgboost, and are comfortable using 100-800 variables in our models. I havnt encountered anything to say that using this many variables in random forest or xgboost should cause any concern, but we cannot come to an agreement. Even if holdout results are better using 100+ variables, we are still encouraged to use less.  Can anyone provide any information regarding this topic that might help shape our decision making progress?  Thank you,  Nate
123567
Just a thought:  Parsimonious models have always been the default go-to in model selection, but to what degree is this approach outdated? I'm curious about how much our tendency toward parsimony is a relic of a time of abaci and slide rules (or, more seriously, non-modern computers). Today's computing power enables us to build increasingly complex models with ever-greater ability for prediction. As a result of this increasing ceiling in computing power, do we really still need to gravitate toward simplicity?    Sure, simpler models are easier to understand and interpret, but in the age of ever-growing data sets with greater numbers of variables and a shift towards a greater focus on prediction capability, this might no longer even be achievable or necessary.    Thoughts?
6266
If I have a dataset with $n$ observations and $p$ variables (dimensions), and generally $n$ is small ($n=12-16$), and $p$ may range from small ($p = 4-10$) to perhaps much larger ($p= 30-50$).  I remember learning that $n$ should be much larger than $p$ in order to run principal component analysis (PCA) or factor analysis (FA), but it seems like this may not be so in my data. Note that for my purposes I am rarely interested in any principal components past PC2.   Questions:   What are the rules of thumb for minimum sample size when PCA is OK to use, and when it is not? Is it ever OK to use the first few PCs even if $n=p$ or $n&lt;p$?  Are there any references on this? Does it matter if your main goal is to use PC1 and possibly PC2 either:    simply graphically, or  as synthetic variable then used in regression?
98298
Coefficients and significance of lasso/ridge  I had 628 predictors after forming dummy of all categorical variables. When I ran lot many iterations traditional logistic regression iteration, I came across 15 variables that was giving me pretty good model with good ROC, recall &amp; precision(for certain cut-off) values on test data and also all variables were significant(at p&lt;=0.05). But since it took lot of time, I tried using lasso that gave me 50 non-zero-coefficient variables after taking best lambda value post running 10 fold cross-validation. But only 5 variables were common between 15 variables of traditional method and 50 of lasso. Moreover, when I tried to calculate its SE and t-stats, I figured out that many variables are insignificant(low t-stats and high p-value). In addition to it, the AUC for ROC was less than traditional method.The ROC drops even more when I used traditional logistic regression on 50 variables that were result of lasso. Can someone help me understand the dynamics of it and how I will be able to justify the coefficients of lasso model as they are penalized(I have normalized all the variables before using lasso)?
88427
I've read through the following posts that answered the question I was going to ask:      Here's what I've done so far: I compared Logistic Regression to Random Forests and RF outperformed Logistic. Now the medical researchers I work with want to turn my RF results into a medical diagnostic tool. For example:  If you are an Asian Male between 25 and 35, have Vitamin D below xx and Blood Pressure above xx, you have a 76% chance of developing disease xxx.   However, RF doesn't lend itself to simple mathematical equations (see above links). So here's my question: what ideas do you all have for using RF to develop a diagnostic tool (without having to export hundreds of trees).   Here's a few of my ideas:   Use RF for variable selection, then use Logistic (using all possible interactions) to make the diagnostic equation. Somehow aggregate the RF forest into one "mega-tree," that somehow averages the node splits across trees.  Similar to #2 and #1, use RF to select variables (say m variables total), then build hundreds of classification trees, all of which uses every m variable, then pick the best single tree.    Any other ideas? Also, doing #1 is easy, but any ideas on how to implement #2 and #3?
32123
Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest?  The most obvious thing is the importance of the variables, in the simplest variant it can be done just by calculating the number of occurrences of a variable. The second thing I was thinking about are interactions. I think that if the number of trees is sufficiently large then the number of occurrences of pairs of variables can be tested (something like chi square independence).  The third thing are nonlinearities of variables. My first idea was just to look at a chart of a variable  Vs score, but I'm not sure yet whether it makes any sense.    Added 23.01.2012 Motivation    I want to use this knowledge to improve a logit model. I think (or at least I hope) that it is possible to find interactions and nonlinearities that were overlooked.
77958
Hi I'm studying regression techniques.  My data has 15 features and 60 million examples (regression task).  When I tried many known regression techniques (gradient boosted tree, Decision tree regression, AdaBoostRegressor etc) linear regression performed great.   Scored almost best among those algorithms.  What can be the reason for this? Because my data has so many examples so DT based method can fit well.   regularized linear regression ridge, lasso performed worse   Can anyone tell me about other performing-well regression algorithms?   Is Factorization Machine and Support vector regression good regression technique to try?
25641
I know that a neural-network architecture is mostly based on the problem itself and the types of input/output, but still - there's always a "square one" when starting to build one. So my question is - given a input dataset of MxN (M  is the number of records, N is the number of features) and a C possible output classes - is there a thumb-rule to how many layers/units should we start with?
84113
I am trying to find the variable importance on a credit scoring database. I have categorical inputs as well as numerical inputs. My question is does the random forest algorithm works the same way when I   case (1) : give only categorical variables as inputs  case (2): both categorical and numerical variables as inputs.  By this what I am actually asking is only categorical variables are enough or numerical variables should be included along with categorical variables as inputs to the algorithm...???  Thanks in advance...
26147
Can a Random Forest be trained to appropriately predict count data? How would this proceed? I have quite a extensive range of values so classification doesn't really make sense. If I would use regression would I simply truncate the results? I'm quite lost here. Any ideas?
50676
I use Lasso logistic regression in order to identify a smaller subset of important variables. I start with N=51 (28/23) and 32 predictors.   So far it looks pretty promising, because I can identify four important predictors in my optimal model.   Now I would like to take those four predictors and examine them along with some control variables in a standard logistic regression.   My question is, does that analysis strategy make sense? Is there a better way to include controls or other variables that might be interesting?  For a better understanding:   Identify important variables via Lasso logistic regression Do further analysis including identified predictors and other control variables using standard logistic regression (using AIC to check model fit)
32233
I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max 20-30 for later use with model training (binary classification - increase / decrease).  I was thinking about using random forests for feature ranking. Is it a good idea to use them recursively? For example, let's say in the first round I drop the worst 20%, second too and so on until I get the desired number of features.  Should I use cross-validation with RF? (It's intuitive for me not to use CV because that's pretty much what RF does already.)  Also if I go with random forests should I use them as classifiers for the binary or regressor for the actual increase / decrease to get feature importances?  By the way, the models I would like to try after feature selection are: SVM, neural nets, locally weighted regressions, and random forest. I'm mainly working in Python.
82743
Do you ever center AND standardize variables in multiple regression? It seems as if standardization would automatically center variables...is this true?
11877
I'm trying to construct several GLMMs with my thesis data and I can't solve this question. I have data on pollinator abundance and fruit set from plant individuals (with different treatments - related to a pest infection -) sampled at six study sites (spatial replicas) and during two consecutive years (time replicas). The problem is that not all individuals (plants) were sampled in both years (because of lack of bloom). My question is: How should i use &quot;Year&quot;, as random effect or as fixed effect? I constructed three candidate models and compared the AIC: a) Response variable ~Predictor variables + (1 | year/plant) + (1 | site/plant) b) Response variable ~Predictor variables + (1 | year) + (1 | site/plant)  c) Response variable ~Predictor variables * year + (1 | site/plant) Option a) had the lowest AIC and the highest R2, but I read in some papers that is not correct use a variable with less than five levels as random effect ... so, anyone could help me?
39943
I was wondering if there were any machine learning techniques (unsupervised) for modelling longitudinal data? I've always used mixed effects models (mostly non-linear) but I was wondering if there are any other ways of doing this (using machine learning).  By machine learning, I mean random forest, classification/clustering, decision trees and even deep learning, etc.
69179
I want to run a regression with 4 to 5 explanatory variables, but I have only 15 observations. Not being able to assume these variables are normally distributed, is there a non-parametric or any other valid regression method?
55346
I need help thinking about and identifying the kind of regression analysis that would be appropriate for this problem. Nothing I've discovered so far seems quite right. Referrals to articles or examples would be helpful. Thank you.  The data look like this:   The data are observational. The sampling unit is a geographic location (EDIT: let's assume units are independent); I'm just trying to understand the basic analytical problem here). At each sampling unit, there are events of two types: the event type of interest (A) and all other types (B). EDIT: In other words, each event is a binary outcome (success, failure). The outcomes are aggregated to the to the location level (Location 1: Success 3, Failure 2. Location 2: Success 0, Failure 1. Location 3: Success 0, Failure 0. Location 4: Success 4, Failure 9 .... etc.  Often, A=0 and, somewhat less often but still frequently, A+B=0.    I am interested in testing a hypothesis about A, somehow controlling for the total count of events (A+B), so either a proportion A/(A+B) or a count model that controls for the total count.  If I understand correctly, if the counts were large, proportions could be calculated for all units, and I could do beta regression. But that definitely can't happen when the total number of events for a unit is zero.  If all I cared about was the count, I could use a ZIP or other count model (and maybe still can). But the research question regards the frequency of A relative to the total number of events.  But how to control for the total number of events? Does it just go in the predictors of a ZIP or similar model? I suspect it's more complicated than that.  It seems obvious to me that the individual events could be modeled directly using multi-level logistic regression (EDIT: or another model for clustered data), but I'm wondering if there is a simpler way to examine what I'm interested in, and I just somehow haven't seen an example of this.
35328
I understand that when label encoding is used ,the numeric number can be interpreted to have an order and a model could assume a linear relationship. However shouldn't this be a problem when there are in-fact many levels in a categorical variable e.g. country.  How about the case of binary variables which for e.g. instead of gender.male ( 1,0) what if I just used Gender ( 1,0) where gender = 0 is female and 1 is male for instance. This shouldn't impact the model as much as a label encoding to a feature with multiple levels? How would this be for a case where I have a feature three levels ( -1,0,1) where -1 means not applicable, 0 means 'No' and 1 means 'Yes', so instead of having 2 columns feature.not_applicable(1,0) and feature.No(1,0) - mathematically how would models be impacted - models here would be GLMS,  boosting models, Random forests etc. Is label encoding recommended when feature has say &lt;=3 to 4 levels and above that is recommended to do one hot encoding or $n-1$ dummy variables?