Regarding multiple comparisons, could someone please explain me why the power of false discovery rate (FDR) is greater than the power of family-wise error rate (FWER)?
109417
I have read that controlling FDR is less stringent than controlling FWER, such as in :     FDR controlling procedures exert a less stringent control over false discovery compared to familywise error rate (FWER) procedures (such as the Bonferroni correction). This increases power at the cost of increasing the rate of type I errors, i.e., rejecting the null hypothesis of no effect when it should be accepted.   But I was wondering how it is shown to be true mathematically?  Is there some relation between FDR and FWER?
26948
I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I apologise for the wall of text, but I felt it was necessary to provide an overview of my current understanding of these concepts, before I moved on to my actual questions.    From what I've gathered, a $p$-value is simply a measure of surprise, the probability of obtaining a result at least as extreme, given that the null hypothesis is true. Fisher originally intended for it to be a continuous measure.  In the Neyman-Pearson framework, you select a significance level in advance and use this as an (arbitrary) cut-off point. The significance level is equal to the type 1 error rate. It is defined by the long run frequency, i.e. if you were to repeat an experiment 1000 times and the null hypothesis is true, about 50 of those experiments would result in a significant effect, due to the sampling variability. By choosing a significance level, we are guarding ourselves against these false positives with a certain probability. $P$-values traditionally do not appear in this framework.  If we find a $p$-value of 0.01 this does not mean that the type 1 error rate is 0.01, the type 1 error is stated a priori. I believe this is one of the major arguments in the Fisher vs N-P debate, because $p$-values are often reported as 0.05*, 0.01**, 0.001***. This could mislead people into saying that the effect is significant at a certain $p$-value, instead of at a certain significance value.  I also realise that the $p$-value is a function of the sample size. Therefore, it cannot be used as an absolute measurement. A small $p$-value could point to a small, non-relevant effect in a large sample experiment. To counter this, it is important to perform an power/effect size calculation when determining the sample size for your experiment. $P$-values tell us whether there is an effect, not how large it is. See .  My question: How can I reconcile the facts that the $p$-value is a measure of surprise (smaller = more convincing) while at the same time it cannot be viewed as an absolute measurement?  What I am confused about, is the following: can we be more confident in a small $p$-value than a large one? In the Fisherian sense, I would say yes, we are more surprised. In the N-P framework, choosing a smaller significance level would imply we are guarding ourselves more strongly against false positives.   But on the other hand, $p$-values are dependent on sample size. They are not an absolute measure. Thus we cannot simply say 0.001593 is more significant than 0.0439. Yet this what would be implied in Fisher's framework: we would be more surprised to such an extreme value. There's even discussion about the term highly significant being a misnomer:   I've heard that $p$-values in some fields of science are only considered important when they are smaller than 0.0001, whereas in other fields values around 0.01 are already considered highly significant.  Related questions:         (as provided by @amoeba)
109733
If a one-sided Fisher's exact test testing the null hypothesis that A is not superior to B has a p-value of 0.98, is it true that the Fisher's exact test p-value for the null hypothesis that B is not superior to A is 1-0.98 = 0.02?  The question arises from the paper at  where the primary outcome had a one-sided p-value of 0.98, but when I calculated the P value with a one-sided FET, I got a p-value of 0.09. Therefore, wondering why the "other" one-sided FET wouldn't have a p-value of 1-0.09 = 0.91 (instead of 0.98). Would appreciate help in knowing where I'm going wrong.  The Stata command used to generate the p-value of 0.09 was:   csi  2  7 2850 2854 , exact                   |   Exposed   Unexposed  |      Total -----------------+------------------------+------------            Cases |         2           7  |          9         Noncases |      2850        2854  |       5704 -----------------+------------------------+------------            Total |      2852        2861  |       5713                  |                        |             Risk |  .0007013    .0024467  |   .0015754                  |                        |                  |      Point estimate    |    [95% Conf. Interval]                  |------------------------+------------------------  Risk difference |        -.0017454       |   -.0037999    .0003091        Risk ratio |         .2866159       |    .0595926    1.378504   Prev. frac. ex. |         .7133841       |   -.3785042    .9404074   Prev. frac. pop |         .3561301       |                  +-------------------------------------------------                                   1-sided Fisher's exact P = 0.0904                                   2-sided Fisher's exact P = 0.1793
58369
I have a data set that is a group of participants who accept or reject each of two devices, and I'd like to test if the two devices are accepted at different rates. Summary table like so  Overall Acceptance         Accept  Reject X       124     20 Y       111     33   What's the best statistical test to determine if the difference between X's acceptance rate and Y's acceptance rate is significant?  I'm unused to binary data so out of my depth here.
28934
As we all know, there are 2 methods to evaluate the logistic regression model and they are testing very different things  Predictive power: Get a statistic that measures how well you can predict the dependent variable based on the independent variables. The well-known Pseudo R^2 are McFadden (1974) and Cox and Snell (1989).  Goodness-of-fit statistics The test is telling whether you could do even better by making the model more complicated, which is actually testing whether there are any non-linearities or interactions that you have missed.   I implemented both tests on my model, which added quadratic and interaction already:     &gt;summary(spec_q2)      Call:     glm(formula = result ~ Top + Right + Left + Bottom + I(Top^2) +       I(Left^2) + I(Bottom^2) + Top:Right + Top:Bottom + Right:Left,       family = binomial())       Coefficients:                  Estimate Std. Error z value Pr(&gt;|z|)         (Intercept)  0.955431   8.838584   0.108   0.9139         Top          0.311891   0.189793   1.643   0.1003         Right       -1.015460   0.502736  -2.020   0.0434 *       Left        -0.962143   0.431534  -2.230   0.0258 *       Bottom       0.198631   0.157242   1.263   0.2065         I(Top^2)    -0.003213   0.002114  -1.520   0.1285         I(Left^2)   -0.054258   0.008768  -6.188 6.09e-10 ***     I(Bottom^2)  0.003725   0.001782   2.091   0.0366 *       Top:Right    0.012290   0.007540   1.630   0.1031         Top:Bottom   0.004536   0.002880   1.575   0.1153         Right:Left  -0.044283   0.015983  -2.771   0.0056 **      ---     Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1     (Dispersion parameter for binomial family taken to be 1)     Null deviance: 3350.3  on 2799  degrees of freedom     Residual deviance: 1984.6  on 2789  degrees of freedom     AIC: 2006.6  and the predicted power is as below, the MaFadden is 0.4004, and the value between 0.2~0.4 should be taken to present very good fit of the model(Louviere et al (2000), Domenich and McFadden (1975))                                                :  &gt; PseudoR2(spec_q2)     McFadden     Adj.McFadden        Cox.Snell       Nagelkerke McKelvey.Zavoina           Effron            Count        Adj.Count     0.4076315        0.4004680        0.3859918        0.5531859        0.6144487        0.4616466        0.8489286        0.4712500           AIC    Corrected.AIC  2006.6179010     2006.7125925   and the goodness-of-fit statistics:  &gt; hoslem.test(result,phat,g=8)       Hosmer and Lemeshow goodness of fit (GOF) test    data:  result, phat   X-squared = 2800, df = 6, p-value &lt; 2.2e-16  As my understanding, GOF is actually testing the following null and alternative hypothesis:   H0: The models does not need interaction and non-linearity   H1: The models needs interaction and non-linearity  Since my models added interaction, non-linearity already and the p-value shows H0 should be rejected, so I came to the conclusion that my model needs interaction, non-linearity indeed. Hope my interpretation is correct and thanks for any advise in advance, thanks.
97239
OK, here's the situation. Someone posted an  on SO.  There were 2 main answers.   One was stock "you can use this standard data structure - see details on Wiki". Basically, a priority queue.  One was mine, which had a somewhat-original (at least, I don't recall ever reading an official moniker for it) data structure. Actually, two of them, one for each possible domain of data distribution.    BOTH algorithms were faster on average (mods AND look-ups) than the standard Priority Queue when applied to that user's problem domain. The proof, aside from fairly simple O(n) analysis, is that as of today, the OP commented on my post that my data structure was the best he could use and accepted my answer. However, at the time of posting, a couple of people down-voted it initially, with incorrect excuses like "algorithm does not work" (based on incorrectly reading my answer), "Wiki data structure is better" (without reading my O(n) analysis or comprehending it). First down-vote was from a competing Wiki question. Then it seemed to be lemming-like down-voting, including at least 2 serial down-voters that were since cleansed by SO team as per my report.   So, since most of the current down-votes if not all are based on people being obviously technically wrong as acknowledged by original poster, is there some sort of appeals process on SO whereby a technically competent moderator/superuser can review the Q, determine whether the people who down-voted also posted incorrect comments as reasons for down-voting, and fix this?  Please note that this is mostly a question of fairness - at my rep level, the current "-6" rep from the Q are not really that big of a deal (and the main problem was the opportunity cost of people not voting my A up when the thread was active due to seeing negative score and a long thread of comment criticisms - which I can't get back anymore since the thread is no longer active).  And yeah, I'm whining :) But it was a behavior of a couple of people that was grossly unfair and totally contrary to the spirit of SO.
92525
Someone claims that the  adjusted $R^2$ will increase with the addition of an extra variable.  I wonder why, as it is called adjusted (in contrast to the normal $R^2$).   The only condition it has to satisfy (to increase the adjusted $R^2$) is that the F-value (by the way, how is it simple to calculate it?) of the null hypothesis that the new variable is greater than 1.  Can someone give me a hint where the links between the adjusted $R^2$ and the F-Stat of that test are?   And however, who wants to include a new variable in a multiple OLS regression model anyway, if the beta was tested to be 0? Therefore adjusted $R^2$ always changes.
61611
Apparently, as a result of  it appears that Yannis and myself see different statistics when going to the programmers.SE.com migration stats tool and the stackoverflow.com tool respectively.    :        while I see:        Why are the rejection rates so radically different?  Which is correct?  (Or am I simply misinterpreting the data somehow and these are representing different statistics)
42550
1) Why does power gain use a coefficient of 10 unlike the other two?  2) Why can't power gain be negative?
6491
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.   My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).   But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?
2707
I have run proc freq to test association between two binary variables using the Fisher's exact test. However, the outputs seem contradict to each other as Two-sided Pr &lt;= P 0.0355 whereas the OR and the 95% Confidence Limits 0.1097 [0.0022, 1.1380] includes 1.  This means that p-value shows that they are significantly associated while the 95% CI does not indicate that.  My question is that based on the output from SAS, what should I use: p-value or CI?
91696
In the context of a two-sided test, in his book Statistical Issues in Drug Development Stephen Senn writes chapter 13 (Determining sample size), p.201:     if we reject $H_{0}$, the hypothesis which we assert is $H_{1}$, which simply   states that the treatment difference is not zero or, in other words,   that there is a difference between the experimental treatment and   placebo. This is not a very exciting conclusion but it happens to be   the conclusion to which significance in a conventional hypothesis test   leads [...] by observing the sign of the treatment difference, we are   also justified in taking the further step of deciding whether the   treatment is superior or inferior to placebo. A power calculation,   however, merely takes a particular value, $\Delta$, within the range of   possible values of given by $H_{1}$ and poses the question: ‘if this   particular value happens to obtain, what is the probability of coming   to the correct conclusion that there is a difference?’      If we wish to say something about the difference which obtains, then   it is better to quote a so-called ‘point estimate’ of the true   treatment effect, together with associated confidence limits.   From this, I understand that, even if two-sided test alternative hypothesis is non directional, one could determine the direction of the difference by looking at the sign of the estimate difference   But in the previous chapter 12 (One-sided and Two-sided Tests and other Issues to Do with Significance and P-values)p.185, about the statement "Unless we use one-sided tests when comparing a treatment with a placebo, we shall not be entitled to claim superiority of the treatment under any circumstances" he writes:     This argument is not reasonable on two counts (Hauschke and   Steinijans, 1996) [...]   Second, it is obvious that if we test $H_{1\ A}:\mu &gt; 0$ against   $H_{0}:\ \mu = 0$ at size $\alpha/2$ and also $H_{1\ B}:\mu &lt; 0$   against $H_{0}:\ \mu = 0$ at size $\alpha/2$, then since it is   impossible for both hypotheses to be rejected, the probability of   rejecting one or the other given that $H_{0}$ is true is $\alpha/2 + \alpha/2 = \alpha$ . So, by carrying out this procedure we shall   maintain the type I error rate at $\alpha$. Furthermore, we shall   always reject the null hypothesis where the conventional two-sided   test does so, and only reject the hypothesis when this does so. Hence   these two one-sided tests are equivalent to the two-sided test.   Therefore, carrying out a two-sided test and using the results to   determine whether superiority, equivalence or failure to prove a   difference applies is a perfectly acceptable procedure   From that, I understand  that confirmatory directional tests (and hence, multiple comparisons) are needed (and possible) to conclude about the sign of the true difference    $\ $  So here are my questions:   In the context of a significant difference with a two-sided test, is looking at the sign of the point estimate sufficient or not to induce the sign of the true difference?  In the latter case, how does one run a confirmatory directional test in R?   Thanks,   Statistical Issues in Drug Development, 2nd Edition. Stephen Senn © 2007 John Wiley &amp; Sons, Ltd. ISBN: 978-0-470-01877-4 Hauschke D, Steinijans VW (1996) Directional decision for a two-tailed alternative [letter;comment]. Journal of Biopharmaceutical Statistics 6: 211–218.
124238
I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations.   provides a great overview of different classification metrics. Precision and Recall or kappa both seem to be a good choice:     One way to justify the results of such classifiers is by comparing them to those of baseline classifiers and showing that they are indeed better than random chance predictions.    As far as I understand, kappa could be the slightly better choice here, as random chance is taken into account. From  I understand that kappa deals with the concept of information gain:     [...] an Observed Accuracy of 80% is a lot less impressive with an Expected Accuracy of 75% versus an Expected Accuracy of 50% [...]   Therefore, my questions would be:    Is it correct to assume kappa to be a better-suited classification metric for this problem?  Does simply using kappa prevent the negative effects of imbalance on the classification algorithm? Is re-(down/up)-sampling or cost-based learning (see ) still required?
74770
I applied  two different classifiers against the same validation set. It turns out that classifier A is better than classifier B in terms of ROC curve. However, classifier B is better than classifier A in terms of confusion matrix. How to explain this kind of contradiction?
24468
I am trying to understand the benefit of propensity matching over non-parametric regression for causal inference from non-experimental data. As background: the way I understand it, parametric regressions are generally a poor choice for causal inference when selection bias exists. One can try to create a model that takes into account the different baselines between treatment and control groups, but she/he will be be extremely vulnerable to model misspecification.* Matching solves this issue by making the two datasets look &quot;alike.&quot; The exact meaning of alike depends on the matching algorithm used, but all the algorithms strive to eradicate model dependence, with the most common types being stratified and weighted propensity matching. The mathematics for these algorithms are well-developed, and in particular, we know how to extract error bars and statistical significance. What I'm struggling with is why this is superior to just using a non-parametric regression like a decision tree or random forest, which are also designed to prevent model misspecification. After creating the forest, one could run individuals through it assuming treatment or no treatment, and call the difference the estimated treatment effect for that individual. My first guess is that extracting significance, which is critical to causal inference, from trees is difficult, but it seems that statisticians have made strides in that regard over the last decade or so. To be clear, I am not asking about using a tree to develop the propensity scores, but to use one instead of propensity matching. To help kick off the conversation, I've developed five hypotheses for why matching is preferred to non-parametric regression, but haven't been able to find anything proving or disproving any:  Empirical research demonstrates that stratified or weighted propensity matching (the most common types) yield results closer to causal experiments than non-parametric regressions like trees.  Though it is possible to extract significance from non-parametric regressions like random forests, the math isn't settled, or the notion of &quot;significance&quot; for a decision tree variable doesn't map precisely to the notion of &quot;one minus the odds of a type I error.&quot;  Though it is possible to extract significance from non-parametric regressions, the code is difficult to write.  Model misspecification actually is an issue for decision trees due to the tuning required to run them. Empirical observation has demonstrated this is more of an issue for decision trees than it is for matching algorithms.  We don't actually know a lot about whether we can use non-parametric regressions for causal inference, but we do know that matching works, so it there's no reason to reinvent the wheel.   *As detailed in the first ten minutes of this wonderfully intuitive Youtube:
106079
I'm trying to gain an intuition as to why increasing the number of features could reduce performance.  I'm currently using an LDA classifier which performs better bivariately among certain features but worse when looking at more features.  My classification accuracy is performed using a stratified 10-fold xval.    Is there a simple case of when a classifier would work better univariately than bivaraiately to gain a somewhat physical or spatial intuition of what is happening in these higher dimensions?
112414
Recently I have noticed that Type I error of a test is too much persistent. Even if all the values are almost equal (variance is extremely low) and tested to that value (true mean), still there is nearly 5% Type I error of the test. I am frightened and curious to rethink, are the people who have banned (or skeptic about) NHST (null hypothesis significance testing) truly correct? Let's have a look at the following r code:       n=10000 t1err=0 set.seed(1235) for (i in 1:n){    x=rnorm(100, 10, 20)    if (((t.test(x, mu=10))$p.value)&lt;=0.05) (t1err=t1err+1)  } cat("Type I error rate in percentage is", (t1err/n)*100,"%")   It will produce roughly 5% as Type I error. Seems ok. But let's see this:      n=10000   t1err=0   set.seed(1235)   for (i in 1:n){      x=rnorm(100, 10, 0.000001)      if (((t.test(x, mu=10))$p.value)&lt;=0.05) (t1err=t1err+1)    }   cat("Type I error rate in percentage is", (t1err/n)*100,"%")   Each values are extremely close to 10. Still it produces about 5% Type I error. What conclusion we may draw from it? Or, am I missing something?