Issues in calibrating models with multiple unbalanced constraints: the significance of systematic model and data errors

Abstract

Calibrating process-based models using multiple constraints often improves the identifiability of model parameters, helps to avoid several errors compensating each other and produces model predictions that are more consistent with underlying processes. However, using multiple constraints can lead to predictions for some variables getting worse – particularly common when combining data sources with very different sample sizes, an increasingly common situation, for example when combining manual and automated measurements. Here we use a series of simulated virtual data experiments to demonstrate and disentangle the underlying cause of issues that can occur when calibrating models with multiple unbalanced constraints in combination with systematic errors in models and data, proposing a diagnostic tool to help identify whether a calibration is failing due to these factors, and testing the utility of adding terms representing uncertainty in systematic model/data error. We show that unbalanced data by itself is not the problem – when fitting simulated data to the “true” model, we can correctly recover model parameters and the true dynamics of latent variables. However, when there are systematic errors in the model or the data, we cannot recover the correct parameters, and the modelled dynamics of the low data volume variables departs significantly from the true values. We show that representing uncertainty in model structural errors and data biases in the calibration can greatly improve the model fit to low-volume data, and improve coverage of uncertainty estimates. We conclude that the underlying issue is not one of sample size or information content per se, despite the popularity of ad hoc approaches that focus on “weighting” datasets to achieve balance – our results emphasize the importance of considering model structural deficiencies and data systematic biases in the calibration of process-based models.