The Minimum Detectable Difference (MDD) concept for establishing trust in nonsignificant results — a critical review
Abstract
Current regulatory guidelines for pesticide risk assessment recommend that nonsignificant results should be complemented by the minimum detectable difference (MDD), a statistical indicator used to decide whether the experiment could have detected biologically relevant effects. We review the statistical theory of the MDD and perform simulations to understand its properties and error rates, comparing its skill in distinguishing between true and false negatives with two alternatives: the minimum detectable effect (MDE), an indicator based on a post hoc power analysis common in medical studies, and confidence intervals (CIs). Our results demonstrate that MDD and MDE only differ in that the power of the MDD depends on the sample size, and that although both have some skill in distinguishing between false negatives and true absence of an effect, they do not perform as well as using CI upper bounds to establish trust in a nonsignificant result – because, unlike the CI, neither MDD nor MDE consider the estimated effect size in their calculation. We conclude that although MDDs are useful, CIs are preferable for deciding whether to treat a nonsignificant test result as a true negative, or for determining an upper bound for an unknown true effect.
What the paper shows and why it matters (AI-generated)
Regulatory guidance recommends reporting a minimum detectable difference (MDD) alongside any “no significant effect” pesticide study, so reviewers can judge whether the study could even have detected harm. Reviewing the statistic's theory and simulating its error rates, the authors find confidence intervals do the same job better, because unlike the MDD they actually incorporate the effect size that was estimated — a distinction with direct bearing on how protective current pesticide risk assessment really is when field studies are underpowered.