Dispersion tests in generalized linear mixed-effects models: a methods comparison and practical guide for ecologists
Preprint
Abstract
Underdispersion and overdispersion are common issues when analysing ecological data with generalized linear (mixed) models (GLMs/GLMMs). Overdispersion, the phenomenon where observations spread wider than expected by the fitted model, usually leads to anti-conservative p-values and, thus, to inflated type I error. In contrast, underdispersion, a narrower spread of the data than expected, causes overly conservative p-values and, therefore, reduced power. A range of tests has been proposed to detect such dispersion problems, but there are few comparative studies of their performance across models and analysis settings, and, most importantly, sparse recommendations for ecologists on how to check for dispersion issues. Our goal was to identify a general dispersion test for GLMs/GLMMs applicable to standard distributions and random-effects structures commonly used in ecological data analysis. Following an initial review of available tests, we selected two classes of dispersion tests: (1) parametric and nonparametric tests based on Pearson residuals and (2) simulation-based tests that compare the expected and observed residual variance. Comparing their performance by type I error, power, and dispersion estimate, across a range of Poisson and binomial GLMs/GLMMs, we found that the nonparametric Pearson residuals test performed best across all metrics, particularly for data with low incidence or count rates and/or small samples; however, at the cost of high computational expense. The parametric Pearson residuals test, recommended in many books and guidelines, was fast and effective for GLMs, but biased towards underdispersion in GLMMs due to the naïve computation of the random-effect degrees of freedom. The simulation-based residual variance test was slightly less powerful, but showed overall good calibration. The latter offers a compromise between the strengths and weaknesses of the two Pearson-based tests. We conclude that for GLMs, the parametric Pearson residuals test offers the best balance of speed and accuracy. For GLMMs, we recommend either the computationally demanding nonparametric Pearson residuals test or the faster, although somewhat less powerful, simulation-based residual variance test. We also analyze two case studies in ecology that differ in complexity and include recommendations for ecological data analysis to address dispersion issues, using the most commonly used R packages, avoiding pitfalls, and improving model fit and the interpretation of ecological datasets.
What the paper shows and why it matters
(AI-generated)
Ecological counts and proportions are often analysed with generalised linear models (GLMs) and their mixed-effects extension (GLMMs). Data that scatter more widely (overdispersion) or more narrowly (underdispersion) than such a model expects distort p-values. Many dispersion tests exist, but many were designed and checked only for narrow cases such as a simple Poisson GLM. They have rarely been compared across mixed-model structures, so ecologists get little guidance. The popular parametric Pearson test, recommended in many books and guidelines, is reasonable and quick for simple models. But it presupposes a chi-squared null distribution, which needs a known number of residual degrees of freedom (unclear with random effects) and large samples (sparse data break it). The paper asks which dispersion test ecologists can trust across their models.
The paper concludes that no dispersion test excels in every respect: the nonparametric Pearson test is the most reliable general test but computationally costly, so the choice depends on model type, data size and computing time. Simulations of three tests on Poisson and binomial models carry this. The parametric Pearson test is fast and effective for GLMs with enough data, but biased towards underdispersion in GLMMs, because random effects are counted with too few degrees of freedom. The nonparametric Pearson test, which refits the model to simulated data, performs best, particularly for low counts and small samples. The simulation-based residual variance test is well calibrated overall but somewhat less powerful, partly because it is often conservative. In two case studies, overdispersion in Swiss redstart counts traced back to spatial autocorrelation rather than the distribution. In a re-analysis of an orangutan behaviour study, the parametric Pearson test, usable only for overdispersion in GLMMs, could not detect underdispersion that the other tests found.
The paper turns this into a decision guide: the parametric Pearson test for standard GLMs with enough data; the nonparametric Pearson test for GLMMs, or the simulation-based test when refitting is too slow or fails. A significant result does not by itself call for another distribution: first check how far the dispersion statistic departs from 1, then look for misspecification such as spatial autocorrelation. All three tests are available in the DHARMa R package, which two of the authors develop. The simulations focused on overdispersion, and the study is a preprint that has not been peer-reviewed.