I know that decision trees make the split based on some metric such as entropy, information gain, gini index etc. But for continous variables how does it figure the value at which to make a split. For example, my split is on age. How does it find out a specific age and decides values greater than this go to right, and vice versa.? Is it the mean value?
123658
Can anyone please explain how splitting is performed in regression trees when we only have continuous features. I have referred to different papers, but all I could find is formulas or theorems.   Can someone please explain, with an example, how we can build a regression tree from scratch?  That would be a great help.
2680
My data set contains a number of numeric attributes and one categorical. Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr, where CategoricalAttr takes one of three possible values: CategoricalAttrValue1, CategoricalAttrValue2 or CategoricalAttrValue3. I'm using default . It works with numeric data only. So my question: is it correct to split the categorical attribute CategoricalAttr into three numeric (binary) variables, like IsCategoricalAttrValue1, IsCategoricalAttrValue2, IsCategoricalAttrValue3 ?
50273
Is there a utility that split file by newline symbol? e.g if a file contains the following lines,  aa bbb cccc   If I want to split it to 3 files, the desired output would be:  aa, bbb And cccc (in 3 different files)  I already checked the split command, it only cut file by file sizes, not what I want.  If I don't wrote a utility myself, is there any standard tool to use?
91410
I am trying to understand the d-Separation logic in Causal Bayesian Networks. I know how the algorithm works, but I don't exactly understand why the "flow of information" works as stated in the algorithm.    For example in the graph above, lets think that we are only given X and no other variable has been observed. Then according to the rules of d-separation, the information flow from X to D:   X influences A, which is $P(A)\neq P(A|X)$. This is OK, since A causes X and if we know about the effect X, this affects our belief about the cause A. Information flows. X influences B,which is $P(B)\neq P(B|X)$. This is OK, since A has been changed by our knowledge about X, the change at A can influence our beliefs about its cause, B, as well. X influences C,which is $P(C)\neq P(C|X)$. This is OK because we know that B is biased by our knowledge about its indirect effect, X, and since B is biased by X, this will influence B's all direct and indirect effects. C is a direct effect of B and it is influenced by our knowledge about X.   Well, up to this point, everything is OK for me since the flow of the information occurs according to intuitive cause-effect relationships. But I don't get the special behavior of so called "V-structures" or "Colliders" in this scheme. According to the d-Separation theory, B and D are the common causes of C in the graph above and it says that if we did not observe C or any of its descendants, the flow information from X is blocked at C. Well, OK, but my question is why?  From the three steps above, started from X, we saw that C is influenced by our knowledge about X and the information flow occurred according to the cause-effect relationship. The d-Separation theory says that we cannot go from C to D since C is not observed. But I think that since we know that C is biased and D is a cause of C, D should be affected as well while the theory says the opposite. I am clearly missing something in my thinking pattern but can't see what it is.  So I need an explanation of why the flow of information blocked at C, if C is not observed.
81244
I have a set of polylines which I need to split at varying distances from their starting points.   In general, I'm looking for a something similar to the SPLIT function within the editor tool. There it is possible to specify a certain line length where the split should be applied.   For instance: I have a line A of 3200m length, that I need to split into two separate geometries, one from 0 til 1700m, and one from 1700m til 3200m.      Can this be done in ArcGIS Desktop?   I'm using ArcGIS Desktop 9.3.1. on WinXP with Python 2.7.2.
13985
I have learned about the intuition behind the KL Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The source I am reading goes on to say that the intuitive understanding of 'distance' between these two distributions is helpful, but should not be taken literally because for two distributions $P$ and $Q$, the KL Divergence is not symmetric in $P$ and $Q$.   I am not sure how to understand the last statement, or is this where the intuition of 'distance' breaks down?   I would appreciate a simple, but insightful example.
101154
Suppose I have a discrete set of (finite) data (values are only positive integers) (which always remains discrete whenever the observation/survey is taken), divided into some identical categories . Now, I want to calculate the mean values of some random variable for each category, hence getting a sequence of mean values. Considering mean as a random variable (let's call it $M$), it's taking decimal values also (obviously possible). But I am not sure if $M$ is a discrete or a continuous random variable.   My intuition suggest me that it should be discrete. Since, my original data from which I calculated the sets of mean is discrete so the mean can take finitely many values which I suppose is not the case with a continuous random variable where the mean can take infinitely many values within an interval. So is my intuition right or wrong?  Is it true that a discrete random variable takes finitely many values and continuous R.V. can take infinitely many possible values?
114466
Is there a branch of statistics that deals with data for which exact values are not known, but for each individual, we know either a maximum or minimum bound to the value?  I suspect that my problem stems largely from the fact that I am struggling to articulate it in statistical terms, but hopefully an example will help to clarify:  Say there are two connected populations $A$ and $B$ such that, at some point, members of $A$ may "transition" into $B$, but the reverse is not possible. The timing of the transition is variable, but non-random. For example, $A$ could be "individuals without offspring" and $B$ "individuals with at least one offspring". I am interested in the age this progression occurs but I only have cross-sectional data. For any given individual, I can find out if they belong to $A$ or $B$. I also know the age of these individuals. For each individual in population $A$, I know that the age at transition will be GREATER THAN their current age. Likewise, for members of $B$, I know that the age at transition was LESS THAN their current age. But I don't know the exact values.  Say I have some other factor that I want to compare with the age of transition. For example, I want to know whether an individual's subspecies or body size affects the age of first offspring. I definitely have some useful information that should inform those questions: on average, of the individuals in $A$, older individuals will have a later transition. But the information is imperfect, particularly for younger individuals. And vice versa for population $B$.  Are there established methods to deal with this sort of data? I do not necessarily need a full method of how to carry out such an analysis, just some search terms or useful resources to start me off in the right place!  Caveats: I am making the simplifying assumption that transition from $A$ to $B$ is instantaneous. I am also prepared to assume that most individuals will at some point progress to $B$, assuming they live long enough. And I realise that longitutinal data would be very helpful, but assume that it is not available in this case.  Apologies if this is a duplicate, as I said, part of my problem is that I don't know what I should be searching for. For the same reason, please add other tags if appropriate.  Sample dataset: Ssp indicates one of two subspecies, $X$ or $Y$. Offspring indicates either no offspring ($A$) or at least one offspring ($B$)   age ssp offsp   21   Y     A   20   Y     B   26   X     B   33   X     B   33   X     A   24   X     B   34   Y     B   22   Y     B   10   Y     B   20   Y     A   44   X     B   18   Y     A   11   Y     B   27   X     A   31   X     B   14   Y     B   41   X     B   15   Y     A   33   X     B   24   X     B   11   Y     A   28   X     A   22   X     B   16   Y     A   16   Y     B   24   Y     B   20   Y     B   18   X     B   21   Y     B   16   Y     B   24   Y     A   39   X     B   13   Y     A   10   Y     B   18   Y     A   16   Y     A   21   X     A   26   X     B   11   Y     A   40   X     B    8   Y     A   41   X     B   29   X     B   53   X     B   34   X     B   34   X     B   15   Y     A   40   X     B   30   X     A   40   X     B   Edit: example dataset changed as it wasn't very representative
23713
In the following lectures , they describe a tree algorithm for cost complexity pruning on page 21. It says we apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of $\alpha$. My initial thought was that we have a set of $\alpha$ (i.e. $\alpha \in [0.1, 0.2, 0.3])$. And then we compute the K-fold cross-validation for each set $\alpha$ and choose the $\alpha$ corresponding to the lowest K-fold cross validation.  For reference, the K-fold cross-validation that I am referring to is described on slide 15 in the following .  However, after some thinking and reading; I found there is a theorem that the sequence of subtrees went through by weakest link pruning will contain the subtree $T_\alpha$. Where $T_\alpha$ is the subtree that minimizes the cost complexity criterion  $$ C_\alpha (T) = \sum_{m=1}^{|T|} \sum_{i: x_i \in R_m} (y_i - \hat y_{R_m})^2 + \alpha|T| $$ described on slide 19 in the  slides.  Since this theorem exists I thought it might be plausible that you can deduce a neat mapping of each $\alpha \in \mathbb{R}$ to the optimal subtree $T_\alpha$. Or at least we can see that for a certain interval of $\alpha$s it will correspond to a specific subtree. I found this theorem on page 308 of the  of print 4. It would be great if someone could clarify the algorithm and it would help to know if such a mapping exists.
65227
I'd like to determine the relative importance of sets of variables toward a randomForest classification model in R. The importance function provides the MeanDecreaseGini metric for each individual predictor--is it as simple as summing this across each predictor in a set?  For example:  # Assumes df has variables a1, a2, b1, b2, and outcome rf &lt;- randomForest(outcome ~ ., data=df) importance(rf) # To determine whether the "a" predictors are more important than the "b"s, # can I sum the MeanDecreaseGini for a1 and a2 and compare to that of b1+b2?
104452
I'm looking for optimal binning method (discretization) of a continuous variable with respect to a given response (target) binary variable and with maximum number of intervals as a parameter.  example: I have a set of observations of people with "height" (numeral continuous) and "has_back_pains" (binary) variables. I want do discretize height into 3 intervals (groups) at most with different proportion of people with back aches, so that the algorithm maximizes the difference between the groups (with given restrictions for instance, that each interval has at least x observations).  The obvious solution to this problem would be to use decision trees(a simple one-variable model) , but I can't find any function in R that would have "maximal number of branches" as a parameter - all of them divide the variable into 2 gropus  (&lt;=x and >x). SAS miner has a "maximum branch" parameter but I'm looking for a non commercial solution.  some of my variables have just a few unique values (and could be treated as discrete variables) but I want to discretize them as well into a smaller number of intervals.  The closest solution to my problem is implemented in the smbinning package in R (which relies on ctree function from party package) but it has two drawbacks: it's impossible to set the number of intervals (however, you can find a way around it by changing the p parameter) and it doesn't work when data vector has less than 10 unique values. Anyway, you can see the example output here(Cutpoint and Odds columns are crucial):  Cutpoint CntRec CntGood CntBad CntCumRec CntCumGood CntCumBad PctRec BadRate   Odds  LnOdds     WoE     IV 1   &lt;= 272   9081     169   8912      9081        169      8912 0.1874  0.9814 0.0190 -3.9653 -0.6527 0.0596 2   &lt;= 311   8541     246   8295     17622        415     17207 0.1762  0.9712 0.0297 -3.5181 -0.2055 0.0068 3   &lt;= 335   2986     163   2823     20608        578     20030 0.0616  0.9454 0.0577 -2.8518  0.4608 0.0163 4  Missing  27852    1125  26727     48460       1703     46757 0.5747  0.9596 0.0421 -3.1679  0.1447 0.0129 5    Total  48460    1703  46757        NA         NA        NA 1.0000  0.9649 0.0364 -3.3126  0.0000 0.0956   Oh, I'm fully aware that binning results in information loss and that there are better methods, but I'm going to use it for data visualization and treat those variables as a factor.
16653
Something that really confuses me with how ROC plots are generated is that, according to Wikipedia:  The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.  So the algorithm seems to be: Set a decision threshold for binary classification, e.g., a number between $[0, 1]$ that tells which data is labeled as $1$ or $0$. Run the trained classifier on your dataset. Calculate TP and FP, which is just based on the result of your classification. Then plot a point with coordinate (TP, FP) on a 2D plot. Repeat this process for....(?) many many times, until you get many points. Then linearly interpolate between these points to generate a 2D curve. Do I get this process correct?  So my question is: why isn't this naturally a 3D plot? The threshold parameter seems to be pretty important.  First of all, I cannot tell at any given plot on the ROC curve what the threshold was used. Are the points even plotted according some sorted order of these thresholds? I cannot tell.  Also, shouldn't the threshold be critical at creating your final classifier? The ROC curve should have a point that corresponds to the best threshold. But I don't know what that threshold is on this plot.  Finally, this is not related to the ROC plot per-se but just the entire concept. It seems pretty dumb to me to either use a very large or very small threshold. A decision threshold of $0$ or $1$ would not make any sense in creating a classifier, no?   It just doesn't seem to make sense to neglect this information.
107814
1) How do we decide if we split into 2 subnodes, or more? Or is it always 2?  2) How do we decide what threshold is the cutoff? Specifically, you have a continuous variable, do you do a binary search, or how is the cutoff decided?
110108
I don't understand what is the purpose of the decision tree? The way I see it is, it is a series of if-else. Why don't I just use if-else instead of using a decision tree? It is because it decreases the complexity of my code?  I am still spared from calculating entropy and information gain because there are prebuilt algorithms for them where I just plug in the rules right? (Like ID3)  Why do we use it with machine learning now? Because we don't even have to come up with the rules while before we needed to? The machine learns from the training data and based on the attributes it can predict a result?  Does implementing ML in my code decrease overhead more and it makes my code less complex, more effective, faster?
71301
Why does  not split at every point encountered, even if run in an iterative fashion?  Following the suggestions of a thread here and several others, I made a set of points in order to split polylines at specified intervals. In order to do this, I used linear referencing to turn my lines into routes. Then, I exported the attribute table, divided each length by 5, and set points at multiples of 1/5 the length of the line. I used this as the event table to create a layer of events. So, each line has 4 equally spaced points along it, and I thus ultimately need the 5 segments separated by those points.  Curiously, though, when I try to do the second part "Split Line at Point," it seems that only one point is used on a given line at a time, and I'm not sure how it chooses the point (maybe order in its attribute table?) I could run the tool multiple times, but this doesn't seem to solve the problem either. For example, I could see how it would continue splitting segments as more segments are generated, but on e.g. the 4th run I see that some lines are still only split 3 times, even though there is a point within a line segment that was never split and existed before I ran the tool. I'm not sure how the splits are generated, but in any case, it doesn't seem to do what I want it to do, which is produce a split at "each" point.  What should I do to split lines at ALL points along the line?
33168
I am trying to split a network. I have an adjacency matrix that shows edges between nodes. The weight of the edge depends upon the total number of overlapping items between two nodes/union of items between two nodes.  Next, I am trying to partition this adjacency matrix into sub-networks that show tightness of these sub-networks based on the strength of the connections within them. Right now I am using Networkx python's modularity metric to do this splitting. I am looking for a second opinion. Also, I can keep splitting until the sub-networks further until there are no more sub-networks left.  In addition to that, I am searching for a metric that can tell me whether further splitting this sub-network gives me better information (aka better tighter sub-network than the last one).FYI, this is an unsupervised approach.  Hope this makes sense. If not please fire away your comments/questions and I will edit my response accordingly. Any help is much appreciated.  Rocky
84113
I am trying to find the variable importance on a credit scoring database. I have categorical inputs as well as numerical inputs. My question is does the random forest algorithm works the same way when I   case (1) : give only categorical variables as inputs  case (2): both categorical and numerical variables as inputs.  By this what I am actually asking is only categorical variables are enough or numerical variables should be included along with categorical variables as inputs to the algorithm...???  Thanks in advance...