Statistical significance
A result is called statistically significant when its P value falls at or below a preset cut-off, usually 0.05. This flags data as unusual under the test's assumptions, not that an effect is large or important.
What it means in practice
Most textbooks and studies discuss P values only for the null hypothesis, the assumption of no effect [1]. The P value is the probability that the test statistic would be at least as large as the one observed if every assumption of the statistical model were correct, including the hypothesis being tested, which is usually the null hypothesis [1]. A result is declared statistically significant if P is at or below a cut-off, usually 0.05, and nonsignificant otherwise [1].
The cut-off is meant to be fixed in advance as part of the study design, while the P value is computed from the data as a result of the analysis [1]. A small P value simply flags the data as unusual if all the assumptions used to compute it were correct [1]. To see which effect sizes are relatively compatible with the data, given the model, the place to look is the confidence interval [1].
Why it matters when reading nootropics research
A methods guide on statistical tests lists 25 common misinterpretations of P values, confidence intervals and statistical power [1]. Two of them are especially relevant to supplement research.
First, significant does not mean meaningful. In a large study, very small effects or small violations of the analysis assumptions can produce a statistically significant result [1].
Second, a P value above 0.05 does not show that there is no effect. It only means that no effect is one of many possibilities that are compatible with the data [1]. In the same way, a set of individually nonsignificant trials should not be read as showing that the total evidence supports no effect [1].
The guide also warns that P values depend on how the analysis was chosen. Picking which analyses to present based on their P values can produce small P values even when the hypothesis being tested is true [1].
For these reasons NootropicsWork gives the size of each result and its comparison, prefers pre-registered primary outcomes, and labels secondary and subgroup findings as such.
References
- Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016.