Florian Hartig

← Publications

Equivalence testing in pesticide risk assessment: evaluation and practical guidance for design, analysis and interpretation

Dimitry Wintermantel, Julia Osterman, Magdalena M. Mair & Florian Hartig

Environment International, 110576 (2026)
Cite this
@article{wintermantel2026equivalence,
  author = {Dimitry Wintermantel and Julia Osterman and Magdalena M. Mair and Florian Hartig},
  title = {Equivalence testing in pesticide risk assessment: evaluation and practical guidance for design, analysis and interpretation},
  journal = {Environment International},
  pages = {110576},
  year = {2026},
  doi = {10.1016/j.envint.2026.110576},
}

DOI: 10.1016/j.envint.2026.110576
Citation statistics: 0 (Google Scholar) · 0 (OpenAlex), as of 23 October 2026

Google Scholar citations per year, as of 23 October 2026

Funding

This study was supported through funding by European Union's Horizon Europe research and innovation framework programme (WildPosh) (grant agreement No. 101135238); European Union's Horizon Europe research and innovation framework programme (RestPoll) (grant agreement No. 101082102); Svenska Forkingsrådet Formas (2024-00298); Strategic Research Area 'Biodiversity and Ecosystem Services in a Changing Climate' BECC, funded by the Swedish government.

View article (DOI) Preprint

Abstract

Harmful pesticide effects exceeding specific protection goals (SPG) may go undetected in underpowered studies. Regulatory honeybee field studies have consistently failed to reach the statistical power required under European Food Safety Authority (EFSA) guidance, which may have contributed to the approval of high-risk substances. Therefore, EFSA advised a shift from testing the null hypothesis of ‘no effect’ (difference testing) to equivalence testing. Under this new approach, a pesticide is classified as ‘low risk’ if the null hypothesis that its effect exceeds the SPG can be rejected. For honeybees, the recommended SPG is a colony size reduction below 10 %. Critics have argued that this framework requires excessive experimental effort to demonstrate pesticide safety and proposed an alternative equivalence test defining treatment effects relative to the lower bound of the 90 %-control-group confidence interval. Using simulations mimicking a regulatory honeybee field study, we show that although the two equivalence tests share the same trade-off between false ‘low-risk’ and false ‘high-risk’ classifications, only EFSA’s equivalence test reliably identifies pesticides with effects > SPG at α = 0.2. Our results show that increasing sample sizes beyond current practice is often necessary for reliable regulatory assessment. However, for pesticides with effect sizes ≤ 5 %, sample size requirements remain lower than those implied by the power requirement of EFSA’s former difference-test-based guidance. Moreover, covariate adjustment – through a model term or balanced colony allocation using anticlustering randomisation – can reduce sample size requirements without losing power and thus save costs. Finally, we provide practical guidance for anticlustering randomisation, equivalence testing and their implementation in R.

What the paper shows and why it matters (AI-generated)

Higher-tier pesticide studies have traditionally often treated a non-significant difference between treated and control groups as evidence of safety, although such studies are often too small to detect large harm. For honeybees, the European Food Safety Authority (EFSA 2023) therefore recommended an equivalence test: a pesticide is “low risk” only if colony-size loss is shown to stay below a protection goal of 10%. Hotopp et al. (2024) found that most simulated studies failed it and proposed measuring the loss against the lower confidence bound of the control mean. Their concern, that harmless pesticides should be able to pass, is reasonable. The paper asks whether this modification improves the test or merely loosens it.

The paper argues that the equivalence test recommended by the European Food Safety Authority is a sound basis for higher-tier honeybee risk assessment, because its significance level controls how often a pesticide exceeding the harm threshold is wrongly passed. Simulations based on control data from one regulatory study (Rolke et al. 2016) show that both tests discriminate equally well at matched error rates. With 10 sites per treatment, the modified test passed 48% of simulated studies of a pesticide that reduces colony size by 12%; it needs a nominal significance level of about 2% instead of 20% to match EFSA’s protection. Hotopp et al.’s low pass rates, the paper argues, arose because their simulated studies were half the size of the Rolke et al. study. Harmless pesticides (no true effect) can be shown to be low risk with site numbers close to current practice; pesticides with a 5% effect need about 24–34 sites per treatment for 80% power, fewer than the 29–41 that the simulations imply for the earlier guidance’s power requirement (EFSA 2013). Adjusting for initial colony size (in the model or by balanced anticlustering) saves up to about ten sites.

The equivalence test can remain the basis of honeybee risk assessment; more sites and less variation between colonies serve better than changing it. The paper adds guidance: pre-register analysis plans, and add a second test of whether harm exceeds the threshold, so that “high risk” is concluded only when shown; if neither test is significant, the result reads “high risk not excluded”. It adds that other higher-tier ecotoxicity studies with a control may benefit from these principles.

References

  • EFSA (2023). Revised guidance on the risk assessment of plant protection products on bees (Apis mellifera, Bombus spp. and solitary bees). EFSA Journal 21. doi:10.2903/j.efsa.2023.7989
  • Hotopp et al. (2024). Using equivalence tests in higher tier studies of honey bees under the revised EFSA Bee Guidance—How? Integrated Environmental Assessment and Management 20, 1496–1503. doi:10.1002/ieam.4911
  • Rolke et al. (2016). Large-scale monitoring of effects of clothianidin-dressed oilseed rape seeds on pollinating insects in Northern Germany: effects on honey bees (Apis mellifera). Ecotoxicology 25, 1648–1665. doi:10.1007/s10646-016-1725-8
  • EFSA (2013). Guidance on the risk assessment of plant protection products on bees (Apis mellifera, Bombus spp. and solitary bees). EFSA Journal 11. doi:10.2903/j.efsa.2013.3295