Using the ODD protocol and NetLogo to replicate agent-based models
Abstract
Replicating existing models and their key results not only adds credibility to the original work, it also allows modellers to start model development from an existing approach rather than from scratch. New theory can then be developed by changing the assumptions or scenarios tested, or by carrying out more in-depth analysis of the model. However, model replication can be challenging if the original model description is incomplete or ambiguous. Here we show that the use of standards can facilitate and speed up replication: the ODD protocol for describing models, and NetLogo, an easy-to-learn but powerful software platform and language for implementing agent-based models. To demonstrate the benefits of this approach, we conducted a replication experiment on 18 agent-based models from different disciplines. The researchers doing the replications had no or little previous experience using ODD and NetLogo. Their task was to rewrite the original model description using ODD, implement the model in NetLogo and try to replicate at least one exemplary main result. They were also asked to produce, if time allowed, some initial new results with the replicated model, and to record the total time spent on the replication exercise. Replication was successful for 15 out of 18 models. The time taken varied between 2 and 12 days, with an average of 5 days. ODD helped to systematically scan the original model description, while NetLogo proved easy and quick to learn, but difficult to debug when implementation problems arose. Although most of the models replicated were relatively simple, we conclude that even for more complex models it can be useful to use ODD and NetLogo for replication, at least for developing a prototype to help decide how to proceed with the replicated model. Overall, the use of both, standard approaches such as ODD and easy to learn but powerful software such as NetLogo, can promote coherence and efficiency within and between different models and modelling communities. Imagine if all modellers spoke ODD and NetLogo as a common language or lingua franca.
What the paper shows and why it matters
(AI-generated)
Agent-based models, simulations of many interacting individual agents, are used across ecology and the social sciences. When used to understand real systems, their insights are often case-specific and hard to generalise. The paper sees one obstacle to more general models in modelling practice: building a model from scratch tends to be valued more than starting from an existing one. This preference has grounds: developing a model teaches the modeller about the system, whereas simply re-using someone else’s model risks applying it beyond its purpose and trusting it blindly. The paper proposes a middle way, an idea it takes from Thiele and Grimm (2015): starting new projects by replicating an existing model, that is, rebuilding it from its published description. Replication, however, is often thought to be difficult, because original descriptions are often incomplete or ambiguous. The paper challenges the belief that replication is too hard and slow to be a practical way to begin a modelling project.
The paper argues that two standards, the ODD protocol for describing simulation models and the easy-to-learn NetLogo platform for building agent-based ones, make replicating existing agent-based models easier and faster than is generally believed. To test this, researchers, mostly experienced modellers with little or no experience of ODD and NetLogo, replicated 18 models from different disciplines: they rewrote each description in ODD (a recommended structure for writing up a model), implemented the model in NetLogo and reproduced at least one main result. This worked for 15 of the 18 models; the exercises took between 2 and 12 days, 5 on average. ODD helped to scan the original descriptions systematically and expose gaps in them, whereas NetLogo proved quick to learn but hard to debug (to find and fix errors). Most models were simple; complex ones will take longer.
For relatively simple models like these, the results indicate that replication need not be a large investment: a modeller can start from an existing model, change its assumptions or test new scenarios. For more complex models, the paper judges ODD and NetLogo useful at least for a prototype that helps decide how to proceed. More broadly, the paper presents the two as examples of standards that could make modelling more coherent and efficient across modelling communities.
References
- Thiele & Grimm (2015). Replicating and breaking models: good for you and good for ecology. Oikos 124, 691–696. doi:10.1111/oik.02170