The evidence contained in the P-value is context dependent
Opening paragraph
In a recent opinion article, Muff et al. recapitulated well-known objections to the Neyman-Pearson Null-Hypothesis Significance Testing (NHST) framework and called for reforming our practices in statistical reporting. We agree with them on several important points: the significance threshold P < 0.05 is only a convention, chosen as a compromise between type I and II error rates; transforming the P-value into a dichotomous statement leads to a loss of information; and P-values should be interpreted together with other statistical indicators, in particular effect sizes and their uncertainties. In our view, a lot of progress in reporting results can already be achieved by keeping these three points in mind.
What the paper shows and why it matters
(AI-generated)
Muff et al. (2022a) proposed that results sections stop sorting findings into “significant” and “not significant” and instead describe p-values in a graded language of evidence, from “very strong evidence” for an effect at very small p-values down to “little or no evidence” when p exceeds 0.1. The idea has real appeal: it keeps the information that a yes-or-no verdict throws away, avoids an arbitrary cut-off and reads intuitively. It rests, however, on an assumption: that a given p-value stands for a fixed amount of evidence, whatever study produced it.
Hartig and Barraquand argue that graded evidence labels for p-values mislead, because the evidence a p-value carries depends on the sample size and power of the study behind it. How much a given p-value speaks for or against an effect depends on how often such a value would arise with and without the effect, and that changes with the power of the study, a point Lakens (2022) also raised. The letter makes this concrete with a simulated t-test in which p-values are translated into the probability of an effect via the Bayes factor: with 200 observations the same p-value meant much less support for an effect than with 13, and most p-values above 0.1, which the proposal would label “little or no evidence”, in fact favoured the absence of an effect. A fixed verbal scale would therefore attach the same words to different amounts of evidence, and, the letter expects, most readers would take two results labelled “strong evidence” as equally convincing when they need not be.
What follows for practice is simple. Researchers who want to avoid the significance cut-off can report the p-value as a number, and those who want to express the evidence for an effect should use a measure built for that purpose, such as the Bayes factor, whose limitations are at least well understood. In their joint reply, Muff et al. (2022b) accepted the point in principle but considered it relevant only for very small samples. The argument has since been taken up in work on significance levels that change with sample size (Wulff and Taylor 2024).
References
- Muff et al. (2022a). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution 37, 203–210. doi:10.1016/j.tree.2021.10.009
- Lakens (2022). Why P values are not measures of evidence. Trends in Ecology & Evolution 37, 289–290. doi:10.1016/j.tree.2021.12.006
- Muff et al. (2022b). Joint reply to 'Rewriting results in the language of compatibility' by V. Amrhein and S. Greenland, and to 'The evidence contained in the P-value is context dependent' by F. Hartig and F. Barraquand. Trends in Ecology & Evolution 37, 571–572. doi:10.1016/j.tree.2022.03.007
- Wulff & Taylor (2024). How and why alpha should depend on sample size: A Bayesian-frequentist compromise for significance testing. Strategic Organization 22, 550–581. doi:10.1177/14761270231214429