Is there a test for testing the difference between two confusion matrices?
121830
I want to compare the classification performance of two heuristics: $h_0$ and $h_1$. I have around 700 samples for which I know the output class $c$ and the results of the two heuristics $c_{\textit{h_0}}, c_{\textit{h_1}}$.  That is, for each sample I have $\text{sample}_{\textit{id}}, c, c_{\textit{h_0}}, c_{\textit{h_1}}$.   $h_0$ has around ~71% correct predictions of the output class and $h_1$ has around ~77% correct. What I want to calculate now is whether this performance difference is just by chance or if the prediction performance of $h_1$ is statistical significant better than that of $h_0$.  I have found , but I don't know whether I can just put my data in there like the following matrix to calculate a p-value.  $$   \begin{matrix}                                &amp; h_0 &amp; h_1 \\     \text{correct prediction} &amp; 497 &amp; 539 \\     \text{wrong prediction}   &amp; 203 &amp; 161   \end{matrix} $$  I don't know if I can calculate the p-value like that or if I'm completely on the wrong track. I would really appreciate it if someone could point me in the right direction.
27454
Being a foreign language speaker (or writer and reader to be more exact) I would prefer to see the changes in a side by side diff tool.  The current format with the two above each other doesn't make it easy to recognize the improvements.
84305
Is there any way to compare the fit of quasi-Poission and negative binomial models? If so, can it be done in R?
64236
I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are:   SMOTE: Synthetic Minority Over-sampling TEchnique () ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning ()   I have implemented ADASYN because its adaptive nature and ease to extension to multi-class problems.  My question is how to test the oversampled data produced by ADASYN (or any other oversampling methods). It is not clear in the mentioned two paper how they performed their experiments. There are two scenarios:  1- Oversample the whole dataset, then split it to training and testing sets (or cross validation).  2- After splitting the original dataset, perform oversampling on the training set only and test on the original data test set (could be performed with cross validation).  In the first case the results are much better than without oversampling, but I am concerned if there is overfitting. While in the second case the results are slightly better than without oversampling and much worse than the first case. But the concern with the second case is if all minority class samples goes to the testing set, then no benefit will be achieved with oversampling.  I am not sure if there are any other settings to test such data.
8915
I have data on the relative abundance (0 to 1 ratio) of 5 land covers for different landscapes (analogous to species composition). These landscapes are grouped into three geographic areas and I want to test for differences in the composition of the landscapes between those three areas and between two years (I have the compostion of land covers for 2000 and 2010). For this I thought to make a permanova with a year nested in the geographic region. However, I don't know what distance of dis(similarity) I should use to test for differences between groups. Is it okay to use Bray-Curtis simmilarity index or will Spearman-rank correlation be better? Do you recommend any other index?
14748
I have three probability distributions (see graph). How can I test if they are statistically different from one another in R? Also, how could I test if one distribution (the purple one) is different than the other two combined?  Distributions:
12896
I am using r and have ~400 observations divided into 3 groups.  Group A contains 199 observations, while B contains 173 and C contains 24. I have generated a boxplot of their corresponding measures and would like to compare the 3rd quartile between them to see if they are significantly different. I've included the boxplot here.  I'm not sure what test I should use to compare the 3rd quartile. Does anyone know?
29192
I'm currently facing some issue in determining what statistical test might be best to adopt to determine whether or not my three datasets are significantly different from each other. I have the following datasets of probabilities observed within my Experiment: prob355 = [0.06672, 0.07473, 0.14813, 0.25516]; prob532 = [0.07226, 0.08718, 0.09631, 0.14276]; prob1064 = [0.05979, 0.09020, 0.10891, 0.11307]; Within these datasets, each entry corresponds to a probability obtained under certain experimental conditions such that within the dataset the conditions for prob355(1) ≠ prob355(2) prob355(3) ≠ prob355(4). Between the datasets they have been ordered such that prob355(1) = prob532(1) = prob1064(1). These datasets are accompanied by the following errors/standarddeviation/noise in the observed probabilities: model_error_355 = [0.03255, 0.03986, 0.01321, 0.01928]; model_error_532 = [0.01370, 0.00880, 0.00856, 0.04257]; model_error_1064 = [0.03150, 0.02441, 0.02307, 0.02059]; Within these datasets, each entry corresponds to the same entry as in the prob355, prob532 or prob1064. Basically, what I'm looking for is to compare each of the overall datasets (whilst taking the standard deviation in each value into account) amongst each other and determine if overall these datasets are statistically different from each other. Or if possible I would like to find out which of the values are significantly different from each other. The goal of this comparison for the overall datasets would be prove the hypothesis that these values are not significantly different from one another. Secondly, the goal would be to determine if for each of the individual conditions (eg. prob355(1) = prob532(1) = prob1064(1)) the values are significantly different or not. Currently, I'm using Matlab to run my analysis but I'm not bound to this software to run this analysis as long as the alternative is open acces :) So, If anyone knows a good way of making this comparison I would be very happy to hear it. Thanks!
86149
I have two small data sets (sizes $n_1 = 8$ and $n_2 = 21$) which look like they have significantly different variances. I know very little about the underlying distributions, but it's definitely not safe to assume they're normal or anything nice like that, which rules out the F-test. I'm aware I could use one of the other named tests (Bartlett, Brown–Forsythe, ...) although I'm not currently quite sure what they assume about the population distribution, if anything.  Instead, I've tried my hand at using a permutation test: the null hypothesis is that the two datasets have equal variance, so relabel the data points at random and measure the difference in the two variances of the relabelled sets. Out of 1,000,000 attempts, fewer than 40,000 had a larger difference in variance (&lt;4%).  Is it correct to say that, therefore, the difference in variance of the two data sets is significant at the $p &lt; 0.05$ level? If so, is there a well-recognised name for this kind of test?
21830
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using  a cross validation technique (to avoid over-fitting) with AUC ROC as scoring function to compare the performance of the algorithms, but I am getting a weird results with Random forest and AdbBoost, I have a perfect AUC_ROC score (i.e. =1) despite the fact that the recall(TPR) and FPR of this algorithms are different from 1 and 0 respectively .  def FPR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fp / (fp+tn)     return result def FNR(y_true, y_pred):     tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()     result = fn / (tp+fn)     return result FPR_scorer = make_scorer(FPR) FNR_scorer = make_scorer(FNR)     def get_CrossValResults2(model,cv_rst,bestIndx):     best=pd.DataFrame.from_dict(cv_rst).iloc[bestIndx]     roc=&quot;{:.12f}&quot;.format(best['mean_test_roc_auc'])     acc =&quot;{:.0%}&quot;.format(best['mean_test_accuracy'])     prec =&quot;{:.0%}&quot;.format(best['mean_test_precision'])     rec =&quot;{:.0%}&quot;.format( best['mean_test_recall'])     f1 =&quot;{:.0%}&quot;.format(best['mean_test_f1'])     r2=&quot;{:.2f}&quot;.format(best['mean_test_r2'])     g_mean=&quot;{:.2f}&quot;.format(best['mean_test_gmean'])     pr_auc=&quot;{:.8f}&quot;.format(best['mean_test_pr'])     fnr=&quot;{:.0%}&quot;.format(best['mean_test_fnr'])     fpr=&quot;{:.0%}&quot;.format(best['mean_test_fpr'])     rst = pd.DataFrame([[ model, acc,prec,rec,fpr,fnr,f1,roc,pr_auc,g_mean,r2]],columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])     return rst        cross_val_rst = pd.DataFrame(columns = ['Model', 'Accuracy', 'Precision', 'Recall','FPR','FNR', 'F1-Score','ROC_auc','PR_auc','gmean','r2'])              scoring = {'accuracy':'accuracy','recall':'recall','precision':'precision','fpr':FPR_scorer,'fnr':FNR_scorer,'f1':'f1' ,'roc_auc':'roc_auc','pr':'average_precision','gmean':Gmean_scorer,'r2':'r2'}     param_grid = {'n_estimators': [200],               'max_depth': [80,90],               'min_samples_leaf': [2,3, 4],             'min_samples_split': [2,5,12],             'criterion': [ 'gini'],              'class_weight' : [class_weights], 'n_jobs' : [-1]}  clf = GridSearchCV(RandomForestClassifier(class_weight=class_weights), param_grid, cv=kfold,scoring=scoring,refit=refit)#Fit the model bestmodel = clf.fit(X,Y)       cross_val_rst = cross_val_rst.append(get_CrossValResults2(model='Random Forrest',bestIndx=bestmodel.best_index_,cv_rst=bestmodel.cv_results_),ignore_index=True)
25743
I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible?  Here is the code:  fit5_snow&lt;- lm(Response ~ Stimulus, data=audio_snow) anova(fit5_snow)  &gt; anova(fit5_snow) Analysis of Variance Table  Response: Response           Df Sum Sq Mean Sq F value  Pr(&gt;F)   Stimulus   5  73.79 14.7578  2.6308 0.02929 * Residuals 84 471.20  5.6095                   --- Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1   df&lt;-df.residual(fit5_snow) MSerror&lt;-deviance(fit5_snow)/df  comparison &lt;-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)  &gt; comparison &lt;-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)  Study:  HSD Test for audio_snow$Response   Mean Square Error:  5.609524   audio_snow$Stimulus,  means                  audio_snow.Response   std.err replication snow_dry_leaves            4.933333 0.6208034          15 snow_gravel                6.866667 0.5679258          15 snow_metal                 6.333333 0.5662463          15 snow_sand                  6.733333 0.5114561          15 snow_snow                  7.333333 0.5989409          15 snow_wood                  5.066667 0.7713110          15  alpha: 0.05 ; Df Error: 84  Critical Value of Studentized Range: 4.124617   Comparison between treatments means                                Difference   pvalue sig        LCL      UCL snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658 snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325 snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325 snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325 snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658 snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658 snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658 snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991 snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325 snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325 snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325 snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991 snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325 snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991 snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991
118558
Is there a Stack Exchange site for machine learning questions?
53918
With diff -r  I can do this task, however it takes so long because diff checks file's content. I want something that determine that two files are the same regarding of their size, last modified, etc. But no checking bit by bit the file (for example a video takes sooo long). Is there any other way?
49905
How do  methods compare in terms of model variance and bias?   My question is partly motivated by this thread: . The answer there suggests that models learned with leave-one-out cross-validation have higher variance than those learned with regular $K$-fold cross-validation, making leave-one-out CV a worse choice.  However, my intuition tells me that in leave-one-out CV one should see relatively lower variance between models than in the $K$-fold CV, since we are only shifting one data point across folds and therefore the training sets between folds overlap substantially.  Or going in the other direction, if $K$ is low in the $K$-fold CV, the training sets would be quite different across folds, and the resulting models are more likely to be different (hence higher variance).  If the above argument is right, why would models learned with leave-one-out CV have higher variance?
20056
I need to compare two generalizability (G) coefficients for data that are from two separate populations. G coefficients are a type of intraclass correlation coefficient (ICC).   The literature on statistically comparing G coefficients is sparse by all accounts, so I thought that the best I might do was find out the recommendation for statistically comparing ICCs. I've considered Fisher's Z, but I know ICCs are not normally distributed.
5580
I would like to test the difference in response of two variables to one predictor. Here is a minimal reproducible example.   library(nlme)  ## gls is used in the application; lm would suffice for this example m.set &lt;- gls(Sepal.Length ~ Petal.Width, data = iris,                 subset = Species == "setosa") m.vir &lt;- gls(Sepal.Length ~ Petal.Width, data = iris,                 subset = Species == "virginica") m.ver &lt;- gls(Sepal.Length ~ Petal.Width, data = iris,                 subset = Species == "versicolor")   I can see that the slope coefficients are different:  m.set$coefficients (Intercept) Petal.Width    4.7771775   0.9301727 m.vir$coefficients (Intercept) Petal.Width    5.2694172   0.6508306  m.ver$coefficients (Intercept) Petal.Width     4.044640    1.426365    I have three questions:   How can I test the difference between slopes? How can I test the difference between residual variances? What is a simple, effective way to present these comparisons?    A related question, , suggests re-running the model with a dummy variable to differentiate the slopes, are there options that would allow the use of independent data sets?
13257
I have two different analytical methods that can measure the concentration of a particular molecule in a matrix (for instance measure the amount of salt in water)  The two methods are different, and each has it's own error.  What ways exist to show the two methods are equivalent (or not).  I'm thinking that plotting the results from a number of samples measured by both methods on a scatter graph is a good first step, but are there any good statistical methods ?