I am having some question with the randomForest. I use the "creatFolds" in "caret" package to partition the data into training set and test set. After building the model, I found that the test data has higher AUC than the training data. Is this possible? The following is code. Could anyone help me with this? Thanks a lot!  library(randomForest) library(pROC) library(ROCR) library(caret) # prep training to binary outcome train0 &lt;- iris[iris$Species %in% c('virginica', 'versicolor'),] train0$Species &lt;- droplevels(train0$Species) set.seed(1) index &lt;- createFolds(iris$Species, k = 2, list = TRUE, returnTrain = TRUE) train &lt;- train0[index$Fold1,];train &lt;- na.omit(train) test &lt;- train0[-index$Fold1,];test &lt;- na.omit(test) # build model rfmodel &lt;- randomForest(Species~., data=train, importance=TRUE, ntree=2)  #the train AUC rf_p_train &lt;- predict(rfmodel, type="prob")[,2] rf_pr_train &lt;- prediction(rf_p_train, train$Species) r_auc_train &lt;- performance(rf_pr_train, measure = "auc")@y.values[[1]]  r_auc_train   #0.8098  #the test AUC rf_p_test &lt;- predict(rfmodel, type="prob",newdata = test)[,2] rf_pr_test &lt;- prediction(rf_p_test, test$Species) r_auc_test &lt;- performance(rf_pr_test, measure = "auc")@y.values[[1]]  r_auc_test    #0.956
121542
I trained a model with some algorithms like random forest, logistic regression and so on.   My dataset was split into 80% CV train data (so actually 60% of data to train the model and 20 % for testing with cv). I built my model and now I'm using the last 20% (which I didn't used the whole time) but I'm wondering why is the predictive accuracy better with the validation set than with the training set?    specificity validation: 0.62962963 sensitivity validation: 0.94761905   on the left is the mean, on the right is the standard deviation (+/-) with a 10 times cross validation    specificity_train: 0.55  (+/- 0.10)  recall_train: 0.94   (+/- 0.02)   Did I something wrong? Thought this is the normal / best way to test, if the model is overfitted or not.  Or is it OK due to the standard deviation?
106596
I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.  What is the meaning of Gain, Cover, and Frequency and how do we interpret them?  Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters   Are there any other parameters that can tell me more about feature importances?  From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.  I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.  importance_matrix   Output:             Feature         Gain        Cover    Frequence   1:            xxx 2.276101e-01 0.0618490331 1.913283e-02   2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01   3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01   4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01   5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02  214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05 215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05 216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05 217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05 218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05
116077
I am a little bit confused about the Area Under Curve (AUC) of ROC and the overall accuracy.  Will the AUC be proportional to the overall accuracy? In other words, when we have a larger overall accuracy will we definitely a get larger AUC? Or are they by definition positively correlated?  If they are positively correlated, why do we bother reporting both of them in some publications?  In real case, I performed some classification task and got the results as follows: classifier A got an accuracy 85% and AUC of 0.98 and classifier B got an accuracy of 93% and AUC of 0.92. Question is, which classifier is better? Is it possible to get results similar to these (or do my results indicate a bug in my implementation)?
44026
I just finished reading  discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset.   For example, we have 10 samples in test dataset. 9 samples are positive and 1 is negative. We have a terrible model which predicts everything positive. Thus, we will have a metric that TP = 9, FP = 1, TN = 0, FN = 0.  Then, Precision = 0.9, Recall = 1.0. The precision and recall are both very high, but we have a poor classifier.  On the other hand, TPR = TP/(TP+FN) = 1.0, FPR = FP/(FP+TN) = 1.0. Because the FPR is very high, we can identify that this is not a good classifier.  Clearly, ROC is better than PR on imbalanced datasets. Can somebody explain why PR is better?
72778
I've been using the  in R to build predictive models for classification and regression.  Caret provides a unified interface to tune model hyper-parameters by cross validation or boot strapping.  For example, if you are building a simple 'nearest neighbors' model for classification, how many neighbors should you use?  2? 10? 100? Caret helps you answer this question by re-sampling your data, trying different parameters, and then aggregating the results to decide which yield the best predictive accuracy.  I like this approach because it is provides a robust methodology for choosing model hyper-parameters, and once you've chosen the final hyper-parameters it provides a cross-validated estimate of how 'good' the model is, using accuracy for classification models and RMSE for regression models.  I now have some time-series data that I want to build a regression model for, probably using a random forest. What is a good technique to assess the predictive accuracy of my model, given the nature of the data? If random forests don't really apply to time series data, what's the best way to build an accurate ensemble model for time series analysis?
97397
I am having some trouble with two different implementations of a classification problem giving different results. Me and my college who did the other implementation has narrowed the problem down to the way we calculate the area under the receiver operating characteristic curve (AUC). One solution is derived from a formula appearing at least at one location:    $$AUC_1 = \frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}\mathbf{1}_{p_i&lt;p_j}$$  I have ported the implementation into R and compared it with the result $AUC_2$ from the R-package pROC:  auc1 &lt;- function(p) {   values &lt;- p[,1]   positives &lt;- values[ p[,2]==1 ]   negatives &lt;- values[ p[,2]==0 ]   count &lt;- 0   for ( p in positives ) {     for ( n in negatives ) {       if ( p&gt;n ) {         count &lt;- count + 1       }     }   }   return(count/(length(positives) * length(negatives))) }  auc2 &lt;- function(p) {   library(pROC)   c &lt;- roc(p[,2], p[,1], print.auc=TRUE, ci=F, of="auc")   return (auc(c)) }  predicted &lt;- c(0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1) real      &lt;- c(0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1) p &lt;- cbind(predicted, real)  auc1(p) auc2(p)   Can someone shed some light on why I get:  $$AUC_1 = 0.9090909 \mbox{ and } AUC_2 = 0.9545\mbox{?}$$
110696
Currently i am using RF toolbox on MATLAB for a binary classification Problem  Data Set: 50000 samples  and more than 250 features  So what should be the number of trees and randomly selected feature on each split to grow the trees? can any other parameter greatly affect the results?
30589
I used sklearn to compute roc_auc_score for a dataset of 72 instances. The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this possible? I would think that even one misclassification should have dropped the score to slightly below 1.0.  # Python 3.6.4 # numpy==1.14.3 # scikit-learn==0.19.1 # scipy==1.1.0  from sklearn import metrics import numpy as np  y_true = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]) y_prob = np.array([0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.7, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.9, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.9, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0])  # Show which actuals do not match their expected probabilities for index, (actual, predicted_prob) in enumerate(zip(y_true, y_prob)):   if (actual == 1 and predicted_prob &lt;= 0.5) or (actual == 0 and predicted_prob &gt; 0.5):     print (f'Mismatch at index {index}. Actual={actual}, predicted_prob={predicted_prob}')   rocauc = metrics.roc_auc_score(y_true, y_prob)     print (f'ROCAUC: {rocauc}')  # Outputs: # Mismatch at index 14. Actual=1.0, predicted_prob=0.5 # Mismatch at index 68. Actual=1.0, predicted_prob=0.4 # ROCAUC: 1.0   Then I debugged the score computation itself and looked at the coordinate ROC output.  # In sklearn/metrics/ranking.py, line 271: --&gt; 271         fpr, tpr, tresholds = roc_curve(y_true, y_score,     272                                         sample_weight=sample_weight)  ipdb&gt; fpr array([0.        , 0.        , 0.        , 0.18181818, 1.        ]) ipdb&gt; tpr array([0.33333333, 0.66666667, 1.        , 1.        , 1.        ]) ipdb&gt; tresholds array([0.9, 0.7, 0.4, 0.1, 0. ])  # coords = [] # for x, y in zip(fpr, tpr): #   coords.append((x, y))  ipdb&gt; pp coords [(0.0, 0.3333333333333333),  (0.0, 0.6666666666666666),  (0.0, 1.0),  (0.18181818181818182, 1.0),  (1.0, 1.0)]   All of those coordinates are on x=0 or y=1, meaning ROCAUC is showing 1.0. The only plausible explanation I can think of is that if the alg had more fpr/tpr points, it would show a very tight curve that never reaches x,y = (0, 1), and the ROCAUC would be close to 1, but not exactly 1. Is that a reasonable interpretation or am I missing something?
84113
I am trying to find the variable importance on a credit scoring database. I have categorical inputs as well as numerical inputs. My question is does the random forest algorithm works the same way when I   case (1) : give only categorical variables as inputs  case (2): both categorical and numerical variables as inputs.  By this what I am actually asking is only categorical variables are enough or numerical variables should be included along with categorical variables as inputs to the algorithm...???  Thanks in advance...
11877
I'm trying to construct several GLMMs with my thesis data and I can't solve this question. I have data on pollinator abundance and fruit set from plant individuals (with different treatments - related to a pest infection -) sampled at six study sites (spatial replicas) and during two consecutive years (time replicas). The problem is that not all individuals (plants) were sampled in both years (because of lack of bloom). My question is: How should i use &quot;Year&quot;, as random effect or as fixed effect? I constructed three candidate models and compared the AIC: a) Response variable ~Predictor variables + (1 | year/plant) + (1 | site/plant) b) Response variable ~Predictor variables + (1 | year) + (1 | site/plant)  c) Response variable ~Predictor variables * year + (1 | site/plant) Option a) had the lowest AIC and the highest R2, but I read in some papers that is not correct use a variable with less than five levels as random effect ... so, anyone could help me?
60350
I know that Random forest uses Bagging, but how does ensemble give a single prediction in sklearn Bagging vs Random forest? Also, I do not understand why in Bagging you not need to choose a tree estimator, while not in RF?  other than this, are the other steps the same? wht are main implementation differences in sklearn Bagging vs Random forest?  bonus points if you have "explain like I'm 5 years old" simple example using sklearn please.
25743
I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible?  Here is the code:  fit5_snow&lt;- lm(Response ~ Stimulus, data=audio_snow) anova(fit5_snow)  &gt; anova(fit5_snow) Analysis of Variance Table  Response: Response           Df Sum Sq Mean Sq F value  Pr(&gt;F)   Stimulus   5  73.79 14.7578  2.6308 0.02929 * Residuals 84 471.20  5.6095                   --- Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1   df&lt;-df.residual(fit5_snow) MSerror&lt;-deviance(fit5_snow)/df  comparison &lt;-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)  &gt; comparison &lt;-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)  Study:  HSD Test for audio_snow$Response   Mean Square Error:  5.609524   audio_snow$Stimulus,  means                  audio_snow.Response   std.err replication snow_dry_leaves            4.933333 0.6208034          15 snow_gravel                6.866667 0.5679258          15 snow_metal                 6.333333 0.5662463          15 snow_sand                  6.733333 0.5114561          15 snow_snow                  7.333333 0.5989409          15 snow_wood                  5.066667 0.7713110          15  alpha: 0.05 ; Df Error: 84  Critical Value of Studentized Range: 4.124617   Comparison between treatments means                                Difference   pvalue sig        LCL      UCL snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658 snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325 snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325 snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325 snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658 snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658 snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658 snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991 snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325 snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325 snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325 snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991 snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325 snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991 snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991
77208
I've discovered that cElementTree is about 30 times faster than xml.dom.minidom and I'm rewriting my XML encoding/decoding code. However, I need to output XML that contains CDATA sections and there doesn't seem to be a way to do that with ElementTree.  Can it be done?
123758
I'm very confused, when I read on the wikipedia "From a mathematical standpoint, Rand index is related to the accuracy, but is applicable even when class labels are not used."   and   But when I use in R the rand.index function from fossil package and the Accuracy function from MLmetrics it doesn't give the same answer  &gt; Accuracy(predicted, real) [1] 0.8266667  &gt; rand.index(predicted,real) [1] 0.7114989  Please can someone explain to me the difference between these two and which one should I use?  NB:I use the k-medoids clustering algorithm (function pam from cluster package)
110157
I'm new to data mining and I'm trying to train a decision tree against a data set which is highly unbalanced. However, I'm having problems with poor predictive accuracy.  The data consists of students studying courses, and the class variable is the course status which has two values - Withdrawn or Current.   Age Ethnicity Gender Course  ... Course Status   In the data set there are many more instances which are Current than Withdrawn. Withdrawn instances only accounting for 2% of the total instances.  I want to be able to build a model which can predict the probability that a person will withdraw in the future. However when testing the model against the training data, the accuracy of the model is terrible.  I've had similar issues with decision trees where the data is dominated by one or two classes.  What approach can I use to solve this problem and build a more accurate classifier?
117618
I am trying to classify using randomForest classification in GEE . However, I always get the error User memory limit exceeded, so I assume the number  of the feature are too large. Nevertheless, I saw many developers can classify even bigger dataset, so I am wondering if there is a better option there? The code can be accessed via  var fire= ee.FeatureCollection('users/spatola/recovery_rf_1712').map(function(feature){     var num = ee.Number.parse(feature.get('CLC_311'));     return feature.set('CLC_311', num);   }) var indices= ['EVI_mean','Dnbr','NBR2_mean','NBR2_min','NBR2_max','EVI_min','EVI_max']; var fire1=  fire.filter(ee.Filter.notNull(indices)); var fire2= fire1.select('Class_R','Dnbr','EVI_max','NBR2_max','imageId') var bands= ['NBR2_max', 'EVI_max']; var classProperty= 'Class_R'; var random= fire2.randomColumn('random'); var split_train= 0.8; var split_test= 0.2; var trainPartition= random.filter(ee.Filter.lt('random',split_train)); var testPartition= random.filter(ee.Filter.gte('random',split_test));  var RFtrained_c= ee.Classifier.smileRandomForest({  numberOfTrees: 100,  minLeafPopulation:2,  seed: 123,  }) .setOutputMode('CLASSIFICATION') .train({     features: trainPartition,     inputProperties: bands,     classProperty: classProperty,     }) print(RFtrained_c); var dict =  RFtrained_c.explain(); var test = testPartition.classify(RFtrained_c) print(test.first())
114991
I have a data set with following structure:  a word | number of occurrence of a word in a document | a document id    How can I perform a test for normal distribution in R? Probably it is an easy question but I am a R newbie.