Significance level

Significance level is the probability that the results observed in a study (or more extreme results) could have occurred by chance alone. It is closely associated with type 1 error (see type 1 and type 2 errors): incorrectly rejecting the null hypothesis (false positive result). Based on the distribution of the test statistic used, a p value corresponding to the study results is estimated and compared with the pre-specified significance level, to determine whether or not to reject the study’s null hypothesis. Usually a two-sided test is required, except in the case of non-inferiority studies. A threshold level of 5% (α = 0.05) for statistical significance is commonly used, which corresponds to a 1 in 20 chance that the observed difference is real. This level is based on convention rather than statistical theory. If many separate comparisons are being made in the analysis of a study, it is more likely that a significant result for one of these individual comparisons can occur by chance. In this case a tighter significance threshold (e.g. α = 0.01) may be used or adjustments such as Bonferroni correction may be made to p values associated with achieving the conventional threshold for each comparison.