What does it mean a value is statistically significant and when does it occur? For example, at a 5% level of significance, do p-values above or below 0.05 indicate significance?
88380
I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error.  I understand that p-values are defined as "the probability of obtaining a test statistic at least as extreme as the one we actually observed", while a significance level is just an arbitrary cutoff value to gauge if the p-value is significant or not. Type I error is the error of rejected a null hypothesis that was true. However, I am unsure regarding the difference between significance level and the type I error, are they not the same concept?  For example, assume a very simple experiment where I flip a coin 1000 times and count the number of times it lands on 'heads'. My null hypothesis, H0, is that heads = 500 (unbiased coin). I then set my significance level at alpha = 0.05.  I flip the coin 1000 times and then I calculate the p-value, if the p-value is > 0.05 then I fail to reject the null hypothesis and if the p-value is &lt; 0.05 then I reject the null hypothesis.  Now if I did this experiment repeatedly, each time calculating the p-value and either rejecting or failing to reject the null hypothesis and keeping a count of how many I rejected/failed to reject, then I would end up rejecting 5% of null hypotheses which were in actuality true, is that correct? This is the definition of type I error. Therefore, the significance level in Fisher significance testing is essentially the type I error from Neyman-Pearson hypothesis testing if you performed repeated experiments.  Now as for p-values, if I had gotten a p-value of 0.06 from my last experiment and I did multiple experiments and counted all the ones that I got a p-value from 0 to 0.06, then would I also not have a 6% chance of rejecting a true null hypothesis?
15149
I am preparing a paper in the field of Computer Science.  In order to report test results, we usually run a number of tests and report the average of those tests.  For each test, we generate random data.  Because of the randomness, at some points, the results may come out not as expected.  For instance, a graph may be like:   Usually, one should explain why on points 8, 11 and 12 there is a decrease on the plot. Probably, it is because of that randomness.  Not hand-crafting all the graph, but just manipulating a few points makes the graph acceptable:   Since three weeks or so, I work my ass off and try to figure out why my resulting graph looks like the first one. Sometimes I feel like yielding to temptation and just modify the raw data before I go crazy.  I believe, at this point the title became misleading, so let me make it clear:  I am not seeking an advice on data manipulation. I will not manipulate my data. However, I ask to myself "how the hell this can be detected?"  And now, I don't only ask to myself, but to whole community. How is this detected? For editors, referees out there, have you ever detected something like this?
109344
This is a question regarding the binomial test in R.  What happens if I get a p-value &lt; 0.05, which would generally cause me to reject the null hypothesis that the population value is equal to a certain proportion (in this case 0.75), but the hypothesised proportion to be tested (i.e. the null hypothesis) still lies within the 95% confidence interval?  Do I still reject the hypothesis of equal proportions?  Here is an example R code:  x &lt;- 31  n &lt;- 50 binom.test(x, n, p=0.75)  # p-value = 0.04812 # 95 percent confidence interval: # 0.4717492 0.7534989
104162
What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?
21823
I've got a very small coefficient (-0.04) and R-squared (0.028) but a significant P value (&lt;0.0001). My question is:   Is my result still meaningful? How to interpret it?   The result is from a linear regression model in a big database in R.  The independent variable (B) has more than 200 values, whereas the dependent variable (A) has 13 values.   The potential correlation is below:   So, I ran a linear regression model between A and B and the result is:  &gt;mod1&lt;-lm(A~B)          Call:         lm(formula = A ~ B)           Residuals:              Min      1Q  Median      3Q     Max           -63.174 -11.816  -1.651  10.184 118.001            Coefficients:                        Estimate Std. Error t value Pr(&gt;|t|)              (Intercept) 132.274547   0.303723 435.511  &lt; 2e-16 ***          B            -0.036675   0.009052  -4.052 5.13e-05 ***          ---         Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1          Residual standard error: 18.52 on 8093 degrees of freedom            (123 observations deleted due to missingness)         Multiple R-squared:  0.002024,  Adjusted R-squared:  0.001901          F-statistic: 16.42 on 1 and 8093 DF,  p-value: 5.134e-05   As you can see, the coefficient of B is only -0.03 and adjusted R squared is only 0.1% but with a p value &lt;0.0001. Is my result reasonable and countable? Surely, my database is larger (8000 records) and even a very small effect size will show a significant P value. But how would I interpret this?
120007
I have a medical research in which I have two sets of data    Control group: People who have no ailment. Study group: People who have some particular ailment.   Within each category there are subgroups like age, gender, blood pressure ranges, cholesterol ranges, etc.  Now I got a few observations (around 50) for both control group and study group. I have observation of various parameter like blood pressure, cholesterol level etc. I want to prove statistical significance of these parameter (Blood pressure/cholesterol) in study group. What is the best way to do it? Should i use $t$ distribution or normal distribution?  Also when I calculate the $p$-value, should I compare each subgroup in study group (based on range) with corresponding subgroup in control group or calculate $p$-value as a whole for study vs control group?
71525
P-value is defined the probability of obtaining a test-statistic at least as extreme as what is observed, assuming null-hypothesis is true. In other words,  $$P( X \ge t | H_0 )$$ But what if the test-statistic is bimodal in distribution? does p-value mean anything in this context? For example, I am going to simulate some bimodal data in R:    set.seed(0) # Generate bi-modal distribution bimodal &lt;- c(rnorm(n=100,mean=25,sd=3),rnorm(n=100,mean=100,sd=5))  hist(bimodal, breaks=100)     And let's assume we observe a test statistic value of 60. And here we know from the picture this value is very unlikely. So ideally, I would want a statistic procedure that I use(say, p-value) to reveal this. But if we compute for the p-value as defined, we get a pretty high-p value  observed &lt;- 60  # Get P-value sum(bimodal[bimodal &gt;= 60])/sum(bimodal) [1] 0.7991993   If I did not know the distribution, I would conclude that what I observed is simply by random chance. But we know this is not true.   I guess the question I have is: Why, when computing p-value, do we compute the probability for the values "at least as extreme as" the observed?  And if I encounter a situation like the one I simulated above, what is the alternative solution?
5310
I know roughly and informally what a confidence interval is.  However, I can't seem to wrap my head around one rather important detail:  According to Wikipedia:     A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained.     I've also seen similar points made in several places on this site.  A more correct definition, also from Wikipedia, is:     if confidence intervals are constructed across many separate data analyses of repeated (and possibly different) experiments, the proportion of such intervals that contain the true value of the parameter will approximately match the confidence level   Again, I've seen similar points made in several places on this site.  I don't get it.  If, under repeated experiments, the fraction of computed confidence intervals that contain the true parameter $\theta$ is $(1 - \alpha)$, then how can the probability that $\theta$ is in the confidence interval computed for the actual experiment be anything other than $(1 - \alpha)$?  I'm looking for the following in an answer:   Clarification of the distinction between the incorrect and correct definitions above. A formal, precise definition of a confidence interval that clearly shows why the first definition is wrong. A concrete example of a case where the first definition is spectacularly wrong, even if the underlying model is correct.
119134
I am unable to understand what exactly is critical value.My question here is why do we fail to reject a null hypothesis when the test statistic is within the range of critical value.  What i am trying to ask is what is the property of the critical value that makes the hypothesized value to fall within the range implied by the critical value.  Thanks in advance
92773
How many Decimal Places are Needed For Accuracy to a Given Number of Significant Figures?  For example: $1234/3.141 = 392.8$, but $1234/3.14159$ changes the result to $392.7$, so how do I am using enough digits of $\pi$ for my result to be accurate to a given number of significant figures?
26948
I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I apologise for the wall of text, but I felt it was necessary to provide an overview of my current understanding of these concepts, before I moved on to my actual questions.    From what I've gathered, a $p$-value is simply a measure of surprise, the probability of obtaining a result at least as extreme, given that the null hypothesis is true. Fisher originally intended for it to be a continuous measure.  In the Neyman-Pearson framework, you select a significance level in advance and use this as an (arbitrary) cut-off point. The significance level is equal to the type 1 error rate. It is defined by the long run frequency, i.e. if you were to repeat an experiment 1000 times and the null hypothesis is true, about 50 of those experiments would result in a significant effect, due to the sampling variability. By choosing a significance level, we are guarding ourselves against these false positives with a certain probability. $P$-values traditionally do not appear in this framework.  If we find a $p$-value of 0.01 this does not mean that the type 1 error rate is 0.01, the type 1 error is stated a priori. I believe this is one of the major arguments in the Fisher vs N-P debate, because $p$-values are often reported as 0.05*, 0.01**, 0.001***. This could mislead people into saying that the effect is significant at a certain $p$-value, instead of at a certain significance value.  I also realise that the $p$-value is a function of the sample size. Therefore, it cannot be used as an absolute measurement. A small $p$-value could point to a small, non-relevant effect in a large sample experiment. To counter this, it is important to perform an power/effect size calculation when determining the sample size for your experiment. $P$-values tell us whether there is an effect, not how large it is. See .  My question: How can I reconcile the facts that the $p$-value is a measure of surprise (smaller = more convincing) while at the same time it cannot be viewed as an absolute measurement?  What I am confused about, is the following: can we be more confident in a small $p$-value than a large one? In the Fisherian sense, I would say yes, we are more surprised. In the N-P framework, choosing a smaller significance level would imply we are guarding ourselves more strongly against false positives.   But on the other hand, $p$-values are dependent on sample size. They are not an absolute measure. Thus we cannot simply say 0.001593 is more significant than 0.0439. Yet this what would be implied in Fisher's framework: we would be more surprised to such an extreme value. There's even discussion about the term highly significant being a misnomer:   I've heard that $p$-values in some fields of science are only considered important when they are smaller than 0.0001, whereas in other fields values around 0.01 are already considered highly significant.  Related questions:         (as provided by @amoeba)
75738
In my textbooks,  are defined as:   “Significant figures by definition are the reliable digits in a number that are known with certainty.”  “A significant figure is the one which is known to be reasonably reliable.”    Reliable means giving the same result on successive trials or reliable information can be trusted to be accurate. However, at the end of each of the two definitions it is also written that ‘the last digit of a number is generally considered uncertain in the absence of qualifying information.' For example if the mass of an object is 12.248 gm, the last digit which is ‘8’ is uncertain by plus or minus 0.001 gm. This uncertainty is unreliability in the information. So, the digit ‘8’ is uncertain, and thus according to the definition 1 it is not a significant figure. However, the rules for significant figures say that ‘non-zero digits are all significant’. Due to the last digit ‘8’, if the object was weighed with careful handling minimizing the chances of error, the figures preceding ‘8’ that are (1, 2, 2, 4) are certain and reliable. If the mass of another object is 2 gm, then it is uncertain by plus or minus 1 gm i.e. its mass could be 3 gm or could be 1 gm.  ‘2’ is therefore not certain! What is the reason that significant figures are defined to be the reliable digits in a number? In what sense they are said to be reliable?
83385
I am working with a data set having N around 200,000. In regressions, I am seeing very small significance values &lt;&lt; 0.001 associated with very small effect sizes, e.g. r=0.028.  What I'd like to know is, is there a principled way of deciding an appropriate significance threshold in relation to the sample size? Are there any other important considerations about interpreting effect size with such a large sample?
59227
While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed.  Thus the question: are they formally equivalent? I.e. is rejecting an hypotheses $H_0 = 0$ with a critical value $\alpha$ equivalent to $0$ not belonging to the confidence interval with critical value $\alpha$?
71571
I'm pretty new to statistics and I need your help. I have a small sample, as follows:    H4U   0.269   0.357   0.2   0.221   0.275   0.277   0.253   0.127   0.246   I ran the Shapiro-Wilk test using R:    shapiro.test(precisionH4U$H4U)   and I got the following result:   W = 0.9502, p-value = 0.6921   Now, if I assume the significance level at 0.05 than the p-value is larger then alpha (0.6921 > 0.05) and I cannot reject the null hypothesis about the normal distribution, but does it allow me to say that the sample has a normal distribution?  Thanks!
7151
I have some  gained from survey questions.  In my case they are  responses (Strongly Disagree-Disagree-Neutral-Agree-Strongly Agree).  In my data they are coded as 1-5.  I  don't think means would mean much here, so what basic summary statistics are considered usefull?
50509
Johansson (2011) in "" (here is also ) states that lower $p$-values are often considered as stronger evidence against the null. Johansson implies that people would consider evidence against the null to be stronger if their statistical test outputted a $p$-value of $0.01$, than if their statistical test outputted a $p$-value of $0.45$. Johansson lists four reasons why the $p$-value cannot be used as evidence against the null:        $p$ is uniformly distributed under the null hypothesis and can therefore never indicate evidence for the null.    $p$ is conditioned solely on the null hypothesis and is therefore unsuited to quantify evidence, because evidence is always   relative in the sense of being evidence for or against a hypothesis   relative to another hypothesis.   $p$ designates probability of obtaining evidence (given the null), rather than strength of evidence.   $p$ depends on unobserved data and subjective intentions and therefore implies, given the evidential interpretation, that the   evidential strength of observed data depends on things that did not   happen and subjective intentions.      Unfortunately I cannot get an intuitive understanding from Johansson's article. To me a $p$-value of $0.01$ indicates there is less chance the null is true, than a $p$-value of $0.45$. Why are lower $p$-values not stronger evidence against null?