If I want to use lasso and regression tree respectively to generate the important variable lists. I'm wondering when will they generate lists with a huge difference? Under what signal structure?
55306
Since RF can handle non-linearity but can't provide coefficients, would it be wise to use random forest to gather the most important features and then plug those features into a multiple linear regression model in order to obtain their coefficients?
65227
I'd like to determine the relative importance of sets of variables toward a randomForest classification model in R. The importance function provides the MeanDecreaseGini metric for each individual predictor--is it as simple as summing this across each predictor in a set?  For example:  # Assumes df has variables a1, a2, b1, b2, and outcome rf &lt;- randomForest(outcome ~ ., data=df) importance(rf) # To determine whether the "a" predictors are more important than the "b"s, # can I sum the MeanDecreaseGini for a1 and a2 and compare to that of b1+b2?
12261
As explained in , a linear model can be written in the form:  $$ y = \beta_1 x_{1} + \cdots + \beta_p x_{p} + \varepsilon_i,$$  where $y$ is the response variable and  $x_{i}$ is the $i^{th}$ explanatory variable.  Often with the goal of meeting test assumptions, one can transform the response variable. For example we apply the log function on each $y_i$. Transforming a response variable does NOT equate to doing a GLM.  A GLM can be written in the following form (from the )  $$ g(u) = \beta_1 x_{1} + \cdots + \beta_p x_{p} + \varepsilon_i,$$  where $u$ is just another symbol for $y$ as I understand from page 2 in the course handout. $g()$ is called the link function.  I don't really understand the difference between a GLM and LM with transformed variable from the slides in the course. Can you help me with that?
101315
I am looking at using the lasso as a method for selecting features and fitting a predictive model with a binary target.  Below is some code I was playing with to try out the method with regularized logistic regression.  My question is I get a group of "significant" variables but am I able to rank order these to estimate relative importance of each? Can the coefficients be standardized for this purpose of rank by absolute value (I understand that they are shown on the original variable scale through the coef function)? If so, how to do so (using the standard deviation of x and y) .  SAMPLE CODE:      library(glmnet)      #data comes from  #http://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)      datasetTest &lt;- read.csv('C:/Documents and Settings/E997608/Desktop/wdbc.data.txt',head=FALSE)   #appears to use the first level as the target success    datasetTest$V2&lt;-as.factor(ifelse(as.character(datasetTest$V2)=="M","0","1"))   #cross validation to find optimal lambda #using the lasso because alpha=1      cv.result&lt;-cv.glmnet(                      x=as.matrix(dataset[,3:ncol(datasetTest)]),               y=datasetTest[,2],                       family="binomial",                       nfolds=10,                       type.measure="deviance",                      alpha=1                     )  #values of lambda used      histogram(cv.result$lambda)  #plot of the error measure (here was deviance) #as a CI from each of the 10 folds #for each value of lambda (log actually)      plot(cv.result)   #the mean cross validation error (one for each of the #100 values of lambda      cv.result$cvm  #the value of lambda that minimzes the error measure #result: 0.001909601      cv.result$lambda.min     log(cv.result$lambda.min)  #the value of lambda that minimzes the error measure #within 1 SE of the minimum #result: 0.007024236      cv.result$lambda.1se  #the full sequence was fit in the object called cv.result$glmnet.fit #this is same as a call to it directly. #here are the coefficients from the min lambda      coef(cv.result$glmnet.fit,s=cv.result$lambda.1se)
30105
I am new to Machine Learning, and am trying to learn it on my own. Recently I was reading through  and had a basic question.   Slide 13 says that "Least Square Estimate is same as Maximum Likelihood Estimate under a Gaussian model". It seems like it is something simple, but I am unable to see this. Can someone please explain what is going on here? I am interested in seeing the Math.   I will later try to see the probabilistic viewpoint of Ridge and Lasso regression also, so if there are any suggestions that will help me, that will be much appreciated also.
108266
Suppose we have a response Y and predictors X1,....,Xn. If we were to try to fit Y via a linear model of X1,....,Xn, and it just so happened that the true relationship between Y and X1,...,Xn wasn't linear, we might be able to fix the model by transforming the X's somehow and then fitting the model. Moreover, if it just so happened that the X1,...,XN didn't affect y independent of the other features, we also might be able to improve the model by including interaction terms, x1*x3 or x1*x4*x7 or something of the like. So in the linear case, interaction terms might bring value by fixing non-linearity or independence violations between the response and the features.  However, Random Forests don't really make these assumptions. Is including interaction terms important when fitting a Random Forest? Or will just including the individual terms and choosing appropriate parameters allow Random Forests to capture these relationships?
26147
Can a Random Forest be trained to appropriately predict count data? How would this proceed? I have quite a extensive range of values so classification doesn't really make sense. If I would use regression would I simply truncate the results? I'm quite lost here. Any ideas?
86970
I am just looking to understand how mtry works in random forests.  Please correct me if I am wrong.  When you specify mtry (say 10), it takes 10 random variables from your data set and examines them for one tree.  So the next tree would take 10 more random variables, examine them, so on and so forth until it runs through the ntrees that you specify and then returns the average estimates for the best/most important variables?
84637
If I run a randomForest model, I can then make predictions based on the model.  Is there a way to get a prediction interval of each of the predictions such that I know how "sure" the model is of its answer.  If this is possible is it simply based on the variability of the dependent variable for the whole model or will it have wider and narrower intervals depending on the particular decision tree that was followed for a particular prediction?
8818
In order to solve problems of model selection, a number of methods (LASSO, ridge regression, etc.) will shrink the coefficients of predictor variables towards zero. I am looking for an intuitive explanation of why this improves predictive ability. If the true effect of the variable was actually very large, why doesn't shrinking the parameter result in a worse prediction?
66041
Common data-based variable selection procedures (for example, forward, backward, stepwise, all subsets) tend to yield models with undesirable properties, including:   Coefficients biased away from zero. Standard errors that are too small and confidence intervals that are too narrow. Test statistics and p-values that do not have the advertised meaning. Estimates of model fit which are overly optimistic. Included terms which can be meaningless (e.g., exclusion of lower-order terms).   Yet, variable selection procedures persist.  Given the problems with variable selection, why are these procedures necessary?  What motivates their use?  Some proposals to start the discussion....   The desire for interpretable regression coefficients? (Misguided in a model with many IVs?) Eliminate variance introduced by irrelevant variables? Eliminate unnecessary covariance/redundancies among the independent variables? Reduce the number of parameter estimates (issues of power, sample size)   Are there others?  Are the problems addressed by variable selection techniques more or less important than the problems variable selection procedures introduce?  When should they be used?  When should they not be used?
103922
I took a method to determine the best features for a classifier from somewhere and I wanted to ask if it is reliable or not:  I was trying to, very simply, select the best features for a Logistic Regression classifier.   Using Lasso().coef_ it turned out that everytime I switched the training and testing data indexs (with a random ShuffleSplit, so in a CrossValidation process), the 0 lasso coefs changed. So, to visualize it:   Loop1     random_seed=1 Lasso_coefs_obtained = [0,   0.9, 0,  0,   0.1]  Loop2 random_seed=2 Lasso_coefs_obtained = [0.3, 0,   0,  0.7, 0  ]  Loop3 random_seed=3 Lasso_coefs_obtained = [0.1, 0.8, 0,  0.2, 0.2]   In order to determine if a feature coefficient was really null, I would sum the three coefficients lists and determine the 0 elements (which must be those elements that are 0 in the three lists).  Sum_coefs = [0.4, 1.7, 0,  0.9, 0.3]   In my case, I did this method with 10 loops (I have 10 coefficients lists over 72 variables). After 10 loops, only 55 coefficients remain non-zero. So, I delated 17 features for the classifier.   Is this method reliable or I'm just making up an algorithm with no guarantees?
17126
Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of categorical or count data. In these settings, it is often also desirable or helpful to collapse tables by combining levels of a factor where those levels are not distinguishable in terms of how they interact with other factors. Two questions:   Is there a way to use penalized models such as LASSO or elastic net to test for the collapsibility of levels within each factor? If the answer to the first question is yes, can, and should, this be set up in such a way that the collapse of levels and the estimation of model coefficients occurs in a single step?
47919
For the LASSO (and other model selecting procedures) it is crucial to rescale the predictors.  The    is simply to use a 0 mean, 1 standard deviation normalization for continuous variables. But what is there to do with dummies?  E.g.  from the same (excellent) summer school I linked to rescales continuous variables to be between 0 and 1 (not great with outliers though), probably to be comparable to the dummies. But even that does not guarantee that the coefficients should be the same order of magnitude, and thus penalized similarly, the key reason for rescaling, no?
26212
I am working with a lasso regression with the glmnet package. I read these threads: ,  and .  Based on the responses I decided that I need to standardize my data before using it. I do have some questions however:   Do I need to standardize the predictors and the responses or only the predictors? I am using the function scale(myData, center = TRUE, scale = TRUE) for building the model, but I am wondering what do I do when I want to do predictions with a test data set. I think I should also standardize and center the test data, but how to I do that? Substracting the mean from the initial (training) dataset and the dividing it by the standard deviation of the initial dataset?  When I get a result do I need to "backscale" it (using the original mean and standard deviation) or do I already get the "final" result?
32123
Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest?  The most obvious thing is the importance of the variables, in the simplest variant it can be done just by calculating the number of occurrences of a variable. The second thing I was thinking about are interactions. I think that if the number of trees is sufficiently large then the number of occurrences of pairs of variables can be tested (something like chi square independence).  The third thing are nonlinearities of variables. My first idea was just to look at a chart of a variable  Vs score, but I'm not sure yet whether it makes any sense.    Added 23.01.2012 Motivation    I want to use this knowledge to improve a logit model. I think (or at least I hope) that it is possible to find interactions and nonlinearities that were overlooked.
4382
From what I know, using lasso for variable selection handles the problem of correlated inputs. Also, since it is equivalent to Least Angle Regression, it is not slow computationally. However, many people (for example people I know doing bio-statistics) still seem to favour stepwise or stagewise variable selection. Are there any practical disadvantages of using the lasso that makes it unfavourable?