Florian Hartig

← Publications

Does model-free forecasting really outperform the true model?

Florian Hartig & Carsten F. Dormann

Proceedings of the National Academy of Sciences, 110(42), E3975 (2013)
Cite this
@article{hartig2013does,
  author = {Florian Hartig and Carsten F. Dormann},
  title = {Does model-free forecasting really outperform the true model?},
  journal = {Proceedings of the National Academy of Sciences},
  volume = {110},
  number = {42},
  pages = {E3975},
  year = {2013},
  doi = {10.1073/pnas.1308603110},
}

DOI: 10.1073/pnas.1308603110
Citation statistics: 31 (Google Scholar) · 17 (OpenAlex), as of 29 October 2026

Google Scholar citations per year, as of 21 October 2026 (yearly counts are fetched separately from the total)

View article (DOI) Preprint

Opening paragraph

Estimating population models from uncertain observations is an important problem in ecology. Perretti et al. observed that standard Bayesian state–space solutions to this problem may provide biased parameter estimates when the underlying dynamics are chaotic (1). Consequently, forecasts based on these estimates showed poor predictive accuracy compared with simple “model-free” methods, which lead Perretti et al. to conclude that “Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data.” However, a simple modification of the statistical methods also suffices to remove the bias and reverse their results.

What the paper shows and why it matters

(AI-generated)

Models serve both to summarise understanding and to predict, and it is often assumed that the two go together: a “more correct” model should forecast better. Perretti et al. (2013) rightly challenged this. For simulated chaotic population dynamics (extremely sensitive to the smallest disturbance), the structurally correct model, fitted statistically to noisy observations, forecast worse than flexible methods that assume no model structure, and they suggested that such “model-free” forecasting may be the most promising way forward. They also pointed to a real problem: the fits can be biased, and such bias could have gone unnoticed with empirical data. The paper takes on the further assumption that the poor forecasts show a weakness of the mechanistic approach, rather than of the way its parameters were estimated.

The paper argues that the apparent advantage of model-free forecasting over the true model, reported by Perretti et al. (2013), is an artefact of biased parameter estimation. Because chaotic trajectories diverge under the smallest noise, the likelihood (how well parameters explain the data) over a long series can favour wrong but stable parameters over the true chaotic ones, a problem already described by Pisarenko and Sornette (2004). For the same reason, even the true model errs more on average over long horizons than a flat forecast at the series mean, so wrong but stable parameters can look better. Using the logistic map (a simple population model that becomes chaotic at high growth rates) from Perretti et al. as the test case, the full-series fit gave growth rates biased towards stable dynamics once the model became chaotic. Fitting short segments instead gave nearly unbiased estimates, fast convergence and forecasts close to those of the true parameters. The short-term error was then lower than the values Perretti et al. reported for the model-free methods (values taken from their study, not re-run).

The results, shown for the logistic map only, give little reason to regard model-free forecasting as generally superior; the paper conjectures, without testing it, that the same holds for the other models. Mechanistic models can therefore keep their other advantages, such as transferability and theoretical understanding, if estimation is adapted to chaotic dynamics. Hamilton et al. (2017) cite the paper as showing that better model fitting can let model-based (parametric) forecasts outperform model-free ones.

References

  • Perretti et al. (2013). Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data. Proceedings of the National Academy of Sciences 110, 5253–5257. doi:10.1073/pnas.1216076110
  • Pisarenko & Sornette (2004). Statistical methods of parameter estimation for deterministically chaotic time series. Physical Review E 69, 036122. doi:10.1103/physreve.69.036122
  • Hamilton et al. (2017). Hybrid modeling and prediction of dynamical systems. PLOS Computational Biology 13, e1005655. doi:10.1371/journal.pcbi.1005655