I am aware how GB classifiers are constructed as regression trees and predictions are made, but not sure how the initial tree and node splitting for it is done.  Can someone please explain how the first tree in GBM  (classifier) is constructed, and how the node splitting criteria for the first tree is decided.  I am not sure, what we are predicting for the first tree (even assuming a constant is initialized, how gradient of loss are computed from the constant).  If mse is the node split criteria, what is it composed of (I mean, squared difference of what values??). Thank you.
121856
For the input training set ${ \{ ({ x }_{ i }{ y }_{ i })\}  }_{ i=1 }^{ n }$  if the loss function is L(y, f(x)), then we initialize the model $M_0$ by finding the $\gamma$ which minimizes: $$ F_0(x) = \sum _{ i=1 }^{ n }{ L{ (y }_{ i, } } \gamma ) $$  which means, for every 'x' we define a model which always gives a constant value $\gamma$  So, now, in the 1st iteration, how come we are able to calculate the derivate of Loss function with respect to the previous model's function, (which is a constant $\gamma$), as derivates with respect to constant are not defined.  Can anyone explain what I'm understanding wrong over here ?
14197
I am performing hierarchical clustering on data I've gathered and processed from the reddit data dump on Google BigQuery.   My process is the following:   Get the latest 1000 posts in /r/politics Gather all the comments Process the data and compute an n x m data matrix (n:users/samples, m:posts/features) Calculate the distance matrix for hierarchical clustering Choose a linkage method and perform the hierarchical clustering Plot the data as a dendrogram   My question is, how do I determine what the best linkage method is? I'm currently using Ward but how do I know if I should be using single, complete, average, etc?  I'm very new to this stuff but I can't find a clear answer online as I'm not sure there is one. So what might be a good idea for my application? Note the data is relatively sparse in the sense that the n x m matrix has a lot of zeroes (most people don't comment on more than a few posts).
35720
I am reading the Pattern Recognition and Machine Learning book by Bishops. I have some difficulties proving the (6.21) equation. Can someone help me?   N.B: If someone has a better title for this question, i'm opened to it as i couldn't find the good words.
73085
Doing research about the xgboost algorithm I went through the .   I have heard that xgboost does not care much about the scale of the input features  In this approach trees are regularized using the complexity definition $$ \Omega(f) = \gamma T + \frac12 \lambda \sum_{j=1}^T w_j^2 $$ where $\gamma$ and $\lambda$ are parameters, $T$ is the number of terminal leaves and $w_j$ is the score in each leaf.  So does this not make it important to scale the features before feeding into xgboost? $\sum_{j=1}^T w_j^2$ term in the regularization part of the cost function is directly influenced by the scale of the features
32233
I have a dataset with mostly financial variables (120 features, 4k examples) which are mostly highly correlated and very noisy (technical indicators, for example) so I would like to select about max 20-30 for later use with model training (binary classification - increase / decrease).  I was thinking about using random forests for feature ranking. Is it a good idea to use them recursively? For example, let's say in the first round I drop the worst 20%, second too and so on until I get the desired number of features.  Should I use cross-validation with RF? (It's intuitive for me not to use CV because that's pretty much what RF does already.)  Also if I go with random forests should I use them as classifiers for the binary or regressor for the actual increase / decrease to get feature importances?  By the way, the models I would like to try after feature selection are: SVM, neural nets, locally weighted regressions, and random forest. I'm mainly working in Python.
73299
When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset.  This is because the models usually have three "levels" of parameters: the first "parameter" is the model class (e.g. SVM, neural network, random forest), the second set of parameters are the "regularization" parameters or "hyperparameters" (e.g. lasso penalty coefficient, choice of kernel, neural network structure) and the third set are what are usually considered the "parameters" (e.g. coefficients for the covariates.)  Given a model class and a choice of hyperparameters, one selects the parameters by choosing the parameters which minimize error on the training set.  Given a model class, one tunes the hyperparameters by minimizing error on the validation set.  One selects the model class by performance on the test set.  But why not more partitions?  Often one can split the hyperparameters into two groups, and use a "validation 1" to fit the first and "validation 2" to fit the second.  Or one could even treat the size of the training data/validation data split as a hyperparameter to be tuned.  Is this already a common practice in some applications?  Is there any theoretical work on the optimal partitioning of data?
64830
I wanted to check if I am understanding the concept of the support vectors correctly. Let's say I am not using any kind of kernel, and it is a hard-margin SVM.  In this case, whatever the number of the dimensions is for our features, the minimum possible number of support vectors equals to 2 (1 for +, 1 for -). Am I correct? Do we still need more support vectors even in such an ideal case in $R^{d}$?
60350
I know that Random forest uses Bagging, but how does ensemble give a single prediction in sklearn Bagging vs Random forest? Also, I do not understand why in Bagging you not need to choose a tree estimator, while not in RF?  other than this, are the other steps the same? wht are main implementation differences in sklearn Bagging vs Random forest?  bonus points if you have "explain like I'm 5 years old" simple example using sklearn please.
84113
I am trying to find the variable importance on a credit scoring database. I have categorical inputs as well as numerical inputs. My question is does the random forest algorithm works the same way when I   case (1) : give only categorical variables as inputs  case (2): both categorical and numerical variables as inputs.  By this what I am actually asking is only categorical variables are enough or numerical variables should be included along with categorical variables as inputs to the algorithm...???  Thanks in advance...
99109
I am a beginner in Machine Learning and I have a situation where an Image needs to be classified into first a super class, then a subclass. For Example, I have a set of images of fruits, containing images of Mangoes, Banana, Peaches, Apples etc. Then each fruit can either be fresh or rotten. I have training set containing images of rotten and fresh fruits for each Fruit type. So for a test image I need to classify which fruit it is (Mango, Banana, etc) and then whether it is fresh or rotten.  I want to know how to approach this kind of problem
115620
I am very confused as to how training data is split and on what data level 0 predictions are made when using generalized stacking. This question is similar to mine, but the answer is not sufficiently clear:    My understanding is that the training set is split, base models trained on one split, and predictions are made on the other. These predictions now become features of a new dataset. One column for each model's prediction, plus a column that contains the ground truth for these predictions.    Split training data into train/test.  Train base models on training split. Make predictions on test split (according to linked answer, use k-fold CV for this). Create a feature for each model, filling it with that model's predictions Create a feature for the ground truth of those predictions. Create a new model and train it on these predictions and ground truth features.   Question 1: Are these the only features used to train the "meta" model? In other words, are none of the actual features of the original data included? The linked answer says it is common to include the original data, but I have not read about it elsewhere.   Question 2: If the above algorithm is correct, What is the form of the data when making predictions? It seems like it would also have to have predictions as independent variables. If so, that means running all new incoming data through all base models again, right?  Question 3: I keep seeing an "out-of-fold" requirement for the first level predictions. It seems that doing a simple train/test split as mentioned above would fulfill this. However, would you not want a 3rd split to test the combined model's generalization? Or is this type of ensemble bulletproof enough not to worry about it?
115366
For statistical and machine learning models, there are multiple levels of interpretability: 1) the algorithm as a whole, 2) parts of the algorithm in general 3) parts of the algorithm on particular inputs, and these three levels split into two parts each, one for training and one for function eval. The last two parts are much closer than to the first. I'm asking about #2, which usually leads to better understanding of #3). (if those are not what 'interpretability' means then what should I be thinking?)  As far as interpretability goes, logistic regression is one of the easiest to interpret. Why did this instance pass the threshold? Because that instance had this particular positive feature and it has a larger coefficient in the model. It's so obvious!  A neural network is  the classic example of a model that is difficult to interpret. What do all those coefficients mean? They all add up in such complicated crazy ways that it is hard to say what any particular coefficient is really doing.  But with all the deep neural nets coming out, it feels like things are becoming clearer. The DL models (for say vision) seem to capture things like edges or orientation in early layers, and in later layers it seems like some nodes are actually semantic (like the proverbial ). For example:    ()  This is a graphic () created by hand for presentation so I am very skeptical. But it is evidence that somebody thinks that is how it works.  Maybe in the past there just weren't enough layers for us to find recognizable features; the models were successful, just not easy to post-hoc analyze particular ones.  But maybe the graphic is just wishful thinking. Maybe NNs are truly inscrutable.  But the many graphics with their nodes labeled with pictures are also really compelling.    Do DL nodes really correspond to features?
56562
I am curious about the practical implementation of a binary split in a decision tree - as it relates to levels of a categorical predictor $X{j}$.  Specifically, I often will utilize some sort of sampling scheme (e.g. bagging, oversampling etc) when building a predictive model using a decision tree - in order to improve its predictive accuracy and stability. During these sampling routines, it is possible for a categorical variable to be presented to a tree fitting algorithm with less than the complete level set.  Say a variable X takes on levels {A,B,C,D,E}. In a sample, maybe only levels {A,B,C,D} are present. Then, when the resulting tree is used for prediction, the full set may be present.  Continuing from this example, say a tree splits on X and sends {A,B} to the left and {C,D} to the right. I would expect the logic of the binary split to then say, when faced with new data: "If X has value A or B, send to the left, otherwise, send this case to the right". What seems to happen in some implementations is "if X has value A or B, send to the left, if X has value C or D send to the right". When this case takes on value E, the algorithm breaks down.   What is the "right" way for a binary split to be handled? It seems the much more robust way is implemented often, but not always (see Rpart below).  Here are a couple examples:  Rpart fails, the others are ok.  #test trees and missing values  summary(solder) table(solder$PadType)  # create train and validation set.seed(12345) t_rows&lt;-sample(1:nrow(solder),size=360, replace=FALSE) train_solder&lt;-solder[t_rows,] val_solder&lt;-solder[-t_rows,]  #look at PadType table(train_solder$PadType) table(val_solder$PadType) #set a bunch to missing levels(train_solder$PadType)[train_solder$PadType %in% c('L8','L9','W4','W9')] &lt;- 'MISSING'   #Fit several trees, may have to play with the parameters to get them to split on the variable  ####RPART mod_rpart&lt;-rpart(Solder~PadType,data=train_solder) predict(mod_rpart,val_solder) #Error in model.frame.default(Terms, newdata, na.action = na.action, xlev = attr(object,  :  #factor 'PadType' has new level(s) D6, L6, L7, L8, L9, W4  ####TREE mod_tree&lt;-tree(Solder~PadType,data=train_solder,split="gini") predict(mod_tree,val_solder) #works fine  ####ctree mod_ctree&lt;-ctree(Solder~PadType,data=train_solder,control = ctree_control(mincriterion = 0.05)) predict(mod_ctree,val_solder) #works fine
94956
I am not an expert of random forest but I clearly understand that the key issue with random forest is the (random) tree generation. Can you explain me how the trees are generated? (i.e. What is the used distribution for tree generation?)  Thanks in advance !
22528
Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in ［0,1］$. Assume  one use gbdt to learn this linear model. And if the test data consists of ($X,Y$) such that $X \in ［4,5］$. Can the learned gbdt model correctly predict data in test set? The reason is that any node in any tree in gbdt has the form: $x_i &gt; a $ and $x_i &lt;=a$, $a$ is the splitting point of variable $x_i$, and the $X$ in test data has disjoint range with the one in train set, so the gbdt model will not be able to distinguish the $X\in ［0,1］$ in train set and $X \in ［4,5］$ in test set, more specifically, the $X\in ［4,5］$ will lie to the right side of the splitting point and then will predict the same value as the $X$ in the upper part of $［0,1］$, for example, in $［2/3,1］$. As the true model is linear, which means it wil not predict the same value for $X$ in train and test set, resulting that the gbdt will not correctly learn such a model. Is the above reasoning correct?
75249
I have a neural network with 3 hidden layers and I'm unsure about the number of hidden nodes for each layer.  Should the number of hidden nodes stay constant between the hidden layers, e.g. 500 nodes for each layer, or should it decrease or decrease?   Is there some rule about how to choose the number of nodes per hidden layer?
2125
This is the objective function for Xgboost.  I have no idea where $g_{i}$ and $h_{i}$ came from  is some one explain how this two terms came form?  or direct me to the related tutorial page then I will search for it