Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Abstract
Ecological data often show temporal, spatial, hierarchical (random effects), or phylogenetic structure. Modern statistical approaches are increasingly accounting for such dependencies. However, when performing cross‐validation, these structures are regularly ignored, resulting in serious underestimation of predictive error. One cause for the poor performance of uncorrected (random) cross‐validation, noted often by modellers, are dependence structures in the data that persist as dependence structures in model residuals, violating the assumption of independence. Even more concerning, because often overlooked, is that structured data also provides ample opportunity for overfitting with non‐causal predictors. This problem can persist even if remedies such as autoregressive models, generalized least squares, or mixed models are used. Block cross‐validation, where data are split strategically rather than randomly, can address these issues. However, the blocking strategy must be carefully considered. Blocking in space, time, random effects or phylogenetic distance, while accounting for dependencies in the data, may also unwittingly induce extrapolations by restricting the ranges or combinations of predictor variables available for model training, thus overestimating interpolation errors. On the other hand, deliberate blocking in predictor space may also improve error estimates when extrapolation is the modelling goal. Here, we review the ecological literature on non‐random and blocked cross‐validation approaches. We also provide a series of simulations and case studies, in which we show that, for all instances tested, block cross‐validation is nearly universally more appropriate than random cross‐validation if the goal is predicting to new data or predictor space, or for selecting causal predictors. We recommend that block cross‐validation be used wherever dependence structures exist in a dataset, even if no correlation structure is visible in the fitted model residuals, or if the fitted models account for such correlations.
What the paper shows and why it matters
(AI-generated)
Cross-validation estimates how well a model will predict new data by fitting it to part of a dataset and testing it on the rest. The usual version holds out a random part, which assumes it is independent of the data used for fitting. Ecological data often break this assumption because observations from nearby places or times, the same group or related species resemble each other. Models that build the structure in (autoregressive, mixed or phylogenetic regressions) are expected to cope with it. Some studies had already validated models on separate blocks of data (Bahn and McGill 2013). The paper asks whether random splits, or models that already include the structure, give trustworthy error estimates for structured data.
The paper compares random and block cross-validation as estimates of prediction error on structured ecological data and argues that random splits underestimate this error, even when the model accounts for the structure. The problem persists with such models because covariates which themselves follow the structure let a model absorb it through non-causal predictors: the residuals look independent, yet predictions fail for new places, times or groups. Block cross-validation, which holds out whole blocks of space, time, individuals or clades, gives estimates closer to the error on independent data. In a simulated landscape analysed with random forests, random splits gave optimistic errors, whereas blocks larger than the range of residual autocorrelation, and leave-one-out validation with a buffer around each test point, came closer to the error on independent data. In a simulated trait–latitude regression with phylogenetic structure, phylogenetic least squares still gave errors that were too low, whereas blocked validation, or leave-one-out validation with a large enough buffer, gave reliable errors and good model selection. Blocks can remove parts of the predictor range from training and so force extrapolation, wanted only if extrapolation is the aim. The paper ties block design to the prediction goal and gives a five-step workflow.
Random cross-validation can be trusted only where independence and the absence of extrapolation hold. The paper recommends block cross-validation wherever dependence exists, even if residuals look independent or the model includes the structure, and argues that a slightly conservative estimate costs little against unwarranted confidence in predictions used for conservation and management. Spatial and environmental blocks can be built with the blockCV R package (Valavi et al. 2019).
References
- Bahn & McGill (2013). Testing the predictive performance of distribution models. Oikos 122, 321–331. doi:10.1111/j.1600-0706.2012.00299.x
- Valavi et al. (2019). block CV : An r package for generating spatially or environmentally separated folds for k ‐fold cross‐validation of species distribution models. Methods in Ecology and Evolution 10, 225–232. doi:10.1111/2041-210x.13107