Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference
Abstract
In ecology, the true causal structure for a given problem is often not known, and several plausible models and thus model predictions exist. It has been claimed that using weighted averages of these models can reduce prediction error, as well as better reflect model selection uncertainty. These claims, however, are often demonstrated by isolated examples. Analysts must better understand under which conditions model averaging can improve predictions and their uncertainty estimates. Moreover, a large range of different model averaging methods exists, raising the question of how they differ in their behaviour and performance. Here, we review the mathematical foundations of model averaging along with the diversity of approaches available. We explain that the error in model‐averaged predictions depends on each model's predictive bias and variance, as well as the covariance in predictions between models, and uncertainty about model weights. We show that model averaging is particularly useful if the predictive error of contributing model predictions is dominated by variance, and if the covariance between models is low. For noisy data, which predominate in ecology, these conditions will often be met. Many different methods to derive averaging weights exist, from Bayesian over information‐theoretical to cross‐validation optimized and resampling approaches. A general recommendation is difficult, because the performance of methods is often context dependent. Importantly, estimating weights creates some additional uncertainty. As a result, estimated model weights may not always outperform arbitrary fixed weights, such as equal weights for all models. When averaging a set of models with many inadequate models, however, estimating model weights will typically be superior to equal weights. We also investigate the quality of the confidence intervals calculated for model‐averaged predictions, showing that they differ greatly in behaviour and seldom manage to achieve nominal coverage. Our overall recommendations stress the importance of non‐parametric methods such as cross‐validation for a reliable uncertainty quantification of model‐averaged predictions.
What the paper shows and why it matters
(AI-generated)
When several plausible models of an ecological system exist and none can be shown to be the true one, ecologists often average their predictions instead of picking a single model. The practice rests on two claims: that the average errs less than the individual models, and that it carries the uncertainty about which model is right into the prediction. Both were often supported by isolated examples. Several earlier reviews covered only weights derived from Akaike’s information criterion (AIC), and no review had covered the many other ways of setting weights and of putting intervals around averaged predictions together. The paper asks when averaging improves predictions, and how weights and uncertainty should be obtained.
Whether averaging improves ecological predictions, this review shows, depends on how the errors of the averaged models are structured. It splits the squared error of an averaged prediction into the models’ bias (systematic error that persists with new data), their variance (scatter of predictions across new datasets), the covariance between their predictions (how far their predictions vary together) and the extra variance from estimating weights from data. Averaging helps mainly when errors are dominated by variance and predictions are only weakly correlated, because then errors partly cancel; a bias that the models share is not averaged away. Estimated weights do not always beat equal weights, although they typically do when many of the models are inadequate. Neither Bayesian nor AIC-based weights are designed to minimise prediction error, and the usual analytical confidence intervals for averaged predictions rarely reach their nominal coverage: they contain the true value less often than the stated percentage promises.
The review concludes that model averaging is a conditional tool, not a default improvement. Its gain should be checked against simple benchmarks, such as the best single model or equal weights. Weights and uncertainty should come from validation on held-out data, through cross-validation or stacking (which chooses weights to minimise the averaged prediction’s error on held-out data), and studies should report the weights and the uncertainty of the predictions – recommendations the authors present as reducing harm rather than as an optimal solution. A later comparison of ensembles of species distribution models found them not consistently better than their untuned component models (Hao et al. 2020).
References
- Hao et al. (2020). Testing whether ensemble modelling is advantageous for maximising predictive performance of species distribution models. Ecography 43, 549–558. doi:10.1111/ecog.04890