I am solving a problem on  binary classification. The Wikipedia  of the Rand index states that for binary classification the Rand index is equivalent to the accuracy: $$ R I=\frac{T P+T N}{T P+F P+F N+T N} $$ Is this statement right?. I am solving a binary classification problem. I have computed the accuracy and the rand index using sklearn.metrics.rand_score. Nevertheless they lead to a different result. I have found a similar question in  but it is not the same case since I have checked that the accuracy doesn't correspond to the Rand neither for the clustering nor for the permutation of the clustering.
123758
I'm very confused, when I read on the wikipedia "From a mathematical standpoint, Rand index is related to the accuracy, but is applicable even when class labels are not used."   and   But when I use in R the rand.index function from fossil package and the Accuracy function from MLmetrics it doesn't give the same answer  &gt; Accuracy(predicted, real) [1] 0.8266667  &gt; rand.index(predicted,real) [1] 0.7114989  Please can someone explain to me the difference between these two and which one should I use?  NB:I use the k-medoids clustering algorithm (function pam from cluster package)
101933
I've looked through a lot of literature available online, including this forum without any luck and hoping someone can help a statistical issue I currently face:  I have 5 lists of of ranked data, each containing 10 items ranked from position 1 (best) to position 10 (worst). For sake of context, the 10 items in each lists are the same, but in different ranked orders as the technique used to decide their rank is different.  Example data:              List 1      List 2      List 3     ... etc Item 1     Ranked 1    Ranked 2    Ranked 1      Item 2     Ranked 3    Ranked 1    Ranked 2 Item 3     Ranked 2    Ranked 3    Ranked 3 ... etc   I am looking for a way to interpret and analyse the above data so that I get a final result showing the overall rank of each item based on each test and its position, e.g.  Result Rank 1 = Item 1 Rank 2 = Item 3 Rank 3 = Item 4 ... etc   So far I have attempted to interpret this information from performing Pearson's Correlation, Spearman's Correlation, Kendall Tau's B, and Friedman tests. I have found however, that these results have generally paired my lists (i.e. compared list 1 to list 2, then list 1 to list 3 .. etc), or have produced results such as Chi-Square, P-Values etc about the overall data.   Does anyone know how I can interpret this data in a statistically sound method (at a post graduate / PhD applicable level) so that I can understand the overall ranks signalling the importance of each item in the list across the 5 tests please? Or, if there is another type of technique or statistical test I can look into I would appreciate any hints or guidance.   (It maybe also worth noting, I have also performed the simpler mathematical techniques such as sums, averaging, minimum - maximum tests etc, but do not feel these are statistically important enough at this level).  Any help or advice would be greatly appreciated, thank you for your time.
25462
Let $\pi_1,...,\pi_n$ be a random permutation of numbers $1,...,n$. If you are told that $\pi_k &gt; \pi_1,...,\pi_k &gt; \pi_{k-1}$, what is the probability that $\pi_k = n$?  What I've tried:  Let $A_j$ be the event that $\pi_k = j$. Let $H$ = be event that $\pi_k &gt; \pi_1,...,\pi_k &gt; \pi_{k-1}$.  Then $P(A_n|H) = \frac{P(H|A_n)P(A_n)}{\sum_{j = 1}^n P(H|A_j)P(A_j)} = \frac{\frac{1}{n}}{\sum_{j=1}^n P(H|A_j)\frac{1}{n}} = \frac{1}{\sum_{j=1}^n P(H|A_j)} = \frac{1}{\sum_{j=1}^n P(j &gt; \pi_1,...,j &gt; \pi_{k-1})}$.  How to calculate that last sum?
34282
I have random variables $X_{1}, X_{2}, \dots X_{n}$ that are each Unif$\{1, \dots, n\}$, and I want to calculate $$P(\{X_{1}, X_{2}, \dots X_{n}\}=\mathbb{N})$$  My intuition is that this means each $X_{n} = n$, which occurs with probability $\frac{1}{n!}$.  Does this seem correct?
96901
I am getting confused at how to calculate the average probability.  Suppose we repeat a kind of binary survey $k$ times each of which was done on a completely separate sample group.  For each $i^{th}$ group, $i=1,2,3,...,k$, Let $n_i$ be the number of samples and $s_i$ be the number of positive results. With this we know that the probability $p_i$ of the positive result for the $i^{th}$ group is $s_i/n_i$  With my ignorance, I happened to use two ways to calculate the average probability in my work wrongly assuming that they are the same:  $$\frac{p_1+p_2+p_3+...+p_k}{k}$$ and $$\frac{s_1+s_2+s_3+...+s_k}{n_1+n_2+n_3+...+n_k}$$  I don't know which is the correct way to calculate the average probability. So could you please explain the difference of these two and when to use which?
84538
I wonder how to compute precision and recall using a confusion matrix for a multi-class classification problem. Specifically, an observation can only be assigned to its most probable class / label. I would like to compute:    Precision = TP / (TP+FP)   Recall = TP / (TP+FN)   for each class, and then compute the micro-averaged F-measure.
27212
I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed the ROC-AUC and the Precision-Recall-AUC (PR-AUC). Then the surprise came!   Method 1 is better than method 2 when I compare them using ROC-AUC. Method 2 is better than method 1 when I compare them using PR-AUC.   Now I'm so confused! How to say which method is better? As far as I know from  that if the ROC-AUC is high, then PR-AUC is also high. So if the ROC curve of method-1 dominates, so should method-1's PR curve. Is my understanding incorrect? Or am I missing something? Because I'm really going crazy!
40408
I am reading about KNN      So I made another example to make things clearer  In this example (Image attached)    You can see there are in total 5 Greed Circles and 20 Blue Squares  by standard KNN (k=3) , X should be a Blue Square   This is obvious 2 Blue Squares vs 1 Green Circle.  But in weighted KNN things are difference  In this case we have to calculate the weight (Likelihood) for each instance  Each Green Circle likelihood is $\frac{1}{5} $ , we have 5 Green Circles  While for Blue Squares it is  $\frac{1}{20} $ , we have 20 Blue Squares  Therefore the weights around X will be $\frac{1}{5} $  Green Circle, and $\frac{2}{20} $ Blue Squares.  which means $\frac{1}{5} &gt; \frac{2}{20} $  Then X is Green Circle  But if try to think of it logically then there are more Blue Squares than Green Circles which means X more likely to be Blue Square than Green Circle.  My question is :  Am I doing anything wrong here? Can someone explain why the equation is showing Green Circle while logic says Blue Square?
28423
Binary data is often mentioned as a nominal sub-category, especially in such examples as female/male, smoker/non-smoker, etc. However, binary data with such values as pass/fail, correct/incorrect, absent/present, etc, seems to give some weight to its values. It's not like in the example of the gender, where both values are equal and differ primarily by the nominal and other context-related traits. Instead, this type of binary data clearly indicates that one value means something and the other means nothing.   In case of such distinction, can binary be considered ordinal? If yes, what are statistical tests that are usually used for such data? Also, are there any interesting books or papers on this case?
27998
Dear everyone - I've noticed something strange that I can't explain, can you? In summary: the manual approach to calculating a confidence interval in a logistic regression model, and the R function confint() give different results.  I've been going through Hosmer &amp; Lemeshow's Applied logistic regression (2nd edition).  In the 3rd chapter there is an example of calculating the odds ratio and 95% confidence interval.  Using R, I can easily reproduce the model:  Call: glm(formula = dataset$CHD ~ as.factor(dataset$dich.age), family = "binomial")  Deviance Residuals:     Min      1Q  Median      3Q     Max   -1.734  -0.847  -0.847   0.709   1.549    Coefficients:                              Estimate Std. Error z value Pr(&gt;|z|)     (Intercept)                   -0.8408     0.2551  -3.296  0.00098 *** as.factor(dataset$dich.age)1   2.0935     0.5285   3.961 7.46e-05 *** --- Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1   (Dispersion parameter for binomial family taken to be 1)      Null deviance: 136.66  on 99  degrees of freedom Residual deviance: 117.96  on 98  degrees of freedom AIC: 121.96  Number of Fisher Scoring iterations: 4   However, when I calculate the confidence intervals of the parameters, I get a different interval to the one given in the text:  &gt; exp(confint(model)) Waiting for profiling to be done...                                  2.5 %     97.5 % (Intercept)                  0.2566283  0.7013384 as.factor(dataset$dich.age)1 3.0293727 24.7013080   Hosmer &amp; Lemeshow suggest the following formula:  $$ e^{[\hat\beta_1\pm z_{1-\alpha/2}\times\hat{\text{SE}}(\hat\beta_1)]} $$   and they calculate the confidence interval for as.factor(dataset$dich.age)1 to be (2.9, 22.9).  This seems straightforward to do in R:  # upper CI for beta exp(summary(model)$coefficients[2,1]+1.96*summary(model)$coefficients[2,2]) # lower CI for beta exp(summary(model)$coefficients[2,1]-1.96*summary(model)$coefficients[2,2])   gives the same answer as the book.  However, any thoughts on why confint() seems to give different results?  I've seen lots of examples of people using confint().
83957
Yesterday's question  got me curious about probability scoring. The  $$\frac{1}{N}\sum\limits _{i=1}^{N}(\text{prediction}_i - \text{reference}_i)^2$$ is a mean squared error measure. Does the analogous mean absolute error performance measure $$\frac{1}{N}\sum\limits _{i=1}^{N}|\text{prediction}_i - \text{reference}_i|$$ have a name, too?
85998
Why is the area under the ROC curve the probability that a classifier will rank a randomly chosen "positive" instance (from the retrieved predictions) higher than a randomly chosen "positive" one (from the original positive class)? How does one prove this statement mathematically using integral, giving the CDFs and PDFs of the true positive and negative class distributions?
47125
Recently, I built a classification model based on the imbalanced data set(positive sample is minority and negative sample is majority), and the model gave the following result for the test set:  True Positives = 0 True Negatives = 139 False Positives = 0 False Negatives = 10.  My question is: for the result, can Matthews correlation coefficient ( ) and  be used for estimating the classifier? Since the denominators for MCC and F-measure are zero, it seems meaningless. If so, MCC and F-measure is not always works for estimating the classifier, and sensitivity and specificity as well as g-mean should be better. Is that right? Any help is appreciated.
95334
Assume we have $N$ random variables $X_1, \ldots, X_N$ with known (posterior) distributions that are easy to sample from. For simplicity, assume that I am interested in the expected value of the ten largest of these random variables. There will be some uncertainty surrounding this expected value as there is uncertainty with respect to the rank of a given $X_i$ in case the distributions of $X_1, \ldots, X_N$ overlap. I want to use simulation based methods. My approach is to simulate $M$ samples from the distributions of $X_1, \ldots, X_N$, where in each iteration, I construct a binary indicator of length $N$ indicating whether random variable $i$ is within the set of the ten largest samples or not. Averaging over these $M$ draws gives me the probability that random variable $i$ is member of the top ten group. Denote this probability as $\pi_i$. Define a binary random variable that indicates membership in the top ten group as $S_i$. $S_i$ follows a Bernoulli distribution s.t. $S_i \sim Ber(\pi_i)$. In a second step, I draw $L$ samples from the distribution of $S_i$ and from the distributions of $X_1, \ldots, X_N$. Given a draw of $S_i$, I can select the top ten out of the samples of $X_1, \ldots, X_N$ in each iteration and average the respective samples within the top ten group. I save this average, so the result after $L$ iterations will be the distribution of the average outcome within the top ten group, taking into account all relevant sources of uncertainty. My questions are: Is this simulation approach valid in general? From a Bayesian perspective, is this simulation valid or do I need to treat additional quantities (e.g. $S_i$ or $\pi_i$) as random variables that need a prior?
1375
I have three groups of data, each with a binomial distribution (i.e. each group has elements that are either success or failure).  I do not have a predicted probability of success, but instead can only rely on the success rate of each as an approximation for the true success rate. I have only found this , which is close but does not seem to exactly deal with the this scenario.  To simplify down the test, let's just say that I have 2 groups (3 can be extended from this base case).   Group 1 trials: $n_1$ = 2455 Group 2 trials: $n_2$ = 2730      Group 1 success: $k_1$ = 1556 Group 2 success: $k_2$ = 1671   I don't have an expected success probability, only what I know from the samples. So my implied success rate for the two groups is:   Group 1 success rate: $p_1$ = 1556/2455 = 63.4%  Group 2 success rate: $p_2$ = 1671/2730 = 61.2%   The success rate of each of the sample is fairly close. However my sample sizes are also quite large. If I check the CDF of the binomial distribution to see how different it is from the first (where I'm assuming the first is the null test) I get a very small probability that the second could be achieved.  In Excel:     1-BINOM.DIST(1556,2455,61.2%,TRUE) = 0.012   However, this does not take into account any variance of the first result, it just assumes the first result is the test probability.  Is there a better way to test if these two samples of data are actually statistically different from one another?
7252
tl;dr  How do I correctly compute the Brier score for more than two classes? I got confusing results with different approaches. Details below.    As suggested to me in a comment to , I would like to evaluate the quality of a set of classifiers I trained with the Brier score. These classifiers are multiclass classifiers and the classes are imbalanced. The Brier score should be able to handle these conditions. However, I am not quite confident about how to apply the Brier score test. Say I have 10 data points and 5 classes:  One hot vectors represent which class is present in a given item of data:  targets = array([[0, 0, 0, 0, 1],                  [0, 0, 0, 0, 1],                  [0, 0, 0, 0, 1],                  [0, 1, 0, 0, 0],                  [0, 0, 0, 0, 1],                  [0, 0, 1, 0, 0],                  [1, 0, 0, 0, 0],                  [0, 1, 0, 0, 0],                  [1, 0, 0, 0, 0],                  [1, 0, 0, 0, 0]])   Vectors of probabilities represent the outputs of my classifiers, assigning a probability to each class  probs = array([[0.14, 0.38, 0.4 , 0.04, 0.05],                [0.55, 0.05, 0.34, 0.04, 0.01],                [0.3 , 0.35, 0.18, 0.09, 0.08],                [0.23, 0.22, 0.04, 0.05, 0.46],                [0.  , 0.15, 0.47, 0.28, 0.09],                [0.23, 0.13, 0.34, 0.27, 0.03],                [0.32, 0.06, 0.59, 0.02, 0.01],                [0.01, 0.19, 0.01, 0.03, 0.75],                [0.27, 0.38, 0.03, 0.12, 0.2 ],                [0.17, 0.45, 0.11, 0.25, 0.01]])   These matrices are coindexed, so probs[i, j] is the probability of class targets[i, j].  Now,  the definition of the Brier Score for multiple classes is  $$\frac{1}{N} \sum_{t=1}^{N} \sum_{i=1}^{R} (f_{ti} - o_{ti})^2$$  When I program this in Python and run it on the above targets and probs matrices, I get a result of $1.0069$  &gt;&gt;&gt; def brier_multi(targets, probs): ...     return np.mean(np.sum((probs - targets)**2, axis=1)) ...  &gt;&gt;&gt; brier_multi(targets, probs) 1.0068899999999998   But I am not sure if I interpreted the definition correctly.  For Python the sklearn library provides . While the documentation states     The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false   What the function actually does is pick one (or get one passed as an argument) of $n &gt; 2$ classes and treat that class as class $1$ and all other classes as class $0$.  For example, if we choose class 3 (index 2) as the $1$ class and thus all other classes as class $0$, we get:  &gt;&gt;&gt; # get true classes by argmax over binary arrays ... true_classes = np.argmax(targets, axis=1) &gt;&gt;&gt;  &gt;&gt;&gt; brier_score_loss(true_classes, probs[:,2], pos_label=2) 0.13272999999999996   alternatively:  &gt;&gt;&gt; brier_score_loss(targets[:,2], probs[:,2]) 0.13272999999999996   This is indeed the , as can be shown by manually defining and running it:  &gt;&gt;&gt; def brier_bin_(targets, probs): ...     return np.mean((targets - probs) ** 2) &gt;&gt;&gt; brier_bin(targets[:,2], probs[:,2]) 0.13272999999999996   As you can see, this is the same result as with sklearn's brier_score_loss.  Wikipedia states about the binary version:     This formulation is mostly used for binary events (for example "rain"   or "no rain"). The above equation is a proper scoring rule only for   binary events;   So... Now I am confused and have the following questions:  1) If sklearn computes the multi class Brier score as a One vs. All binary score, is that the only and correct way to compute the multi class Brier score?  Which leads me to  2) If that is so, my brier_multi code must be based on a misconception. What is my misconception about the definition of the multiclass Brier score?  3) Maybe I am on the wrong track altogether. In which case, please explain to me, how I compute the Brier score correctly?
45834
I'm working through a problem regarding expected values in Markov chains, and at some point it says:     Recall from probability that if $X$ is a positive integer valued random variable, then $\mathbb{E}[X] = 1 + \sum^\infty_{k=1}\mathbb{P}(X &gt; k)$.   I know that by definition $\mathbb{E}[X] = \sum_a a\mathbb{P}(X = a)$, but I can't see how the above equality follows from this, nor am I sure if this is how to approach the problem.