¹¹institutetext: Center for Modeling Social Systems (CMSS),
NORCE Norwegian Research Center AS,
Universitetsveien 19, Kristiansand, Norway
¹¹email: {ongu,timo,paan}@norceresearch.no
https://www.norceresearch.no/en/research-group/cmss

A Guide to Re-Implementing Agent-based Models: Experiences from the HUMAT Model

Önder Gürcan 0000-0001-6982-5658 Timo Szczepanska 0000-0003-2442-8223 Patrycja Antosz 0000-0001-6330-1597

Abstract

Replicating existing agent-based models poses significant challenges, particularly for those new to the field. This article presents an all-encompassing guide to re-implementing agent-based models, encompassing vital concepts such as comprehending the original model, utilizing agent-based modeling frameworks, simulation design, model validation, and more. By embracing the proposed guide, researchers and practitioners can gain a profound understanding of the entire re-implementation process, resulting in heightened accuracy and reliability of simulations for complex systems. Furthermore, this article showcases the re-implementation of the HUMAT socio-cognitive architecture, with a specific focus on designing a versatile, language-independent model. The encountered challenges and pitfalls in the re-implementation process are thoroughly discussed, empowering readers with practical insights. Embrace this guide to expedite model development while ensuring robust and precise simulations.

Keywords:

Agent-based Models Replication Re-implementation Simulation design Model calibration Model validation Best practices

1 Introduction

Recognizing the need to build higher quality social simulation tools, the scientific community has made numerous efforts to develop procedures that improve description [16], reusability [28], rigor and transperency [3], and increase confidence in agent-based model (ABM) outputs. One essential procedure that deserves more attention as an external model validation method is model replication - re-implementing an existing model based on a representation provided by model builders (e.g., in the form of a natural language conceptual description or source code). Even though agent-based modelers have early on recognized replication as "one of the hallmarks of cumulative science" [10] and was proposed, alongside verification and validation, as an independent test of a model’s reliability [33], it is most often brought to attention in negative instances of a failure to reproduce results of the original model (e.g.,[31]).

Since ABMs provide explicit causal explanations of investigated phenomena [9], replication is vital in validating the model’s causal consistency. After all, a causal mechanism represented in the ABM is expected to produce the same effects regardless of the language/software of the implementation. However, if it fails to do so, jumping to conclusions about a widespread replication crisis in social simulation (similar to the one in psychology [22] might be premature, given how much we still have to learn about the specificity of agent-based modeling as a scientific method. Re-implementing a conceptually identical model in a novel platform can help validate the causal mechanisms explaining the model outcomes and identify software-implicit assumptions that are not an explicit part of the conceptual causal explanation but influence the model outcomes (e.g., [14]).

Re-implementing existing ABMs in another programming language is a crucial task for researchers and practitioners seeking to enhance the flexibility and scalability of their simulations. Until now, various studies have emphasized the importance of re-implementing agent-based models in different programming languages [26, 14, 4, 29, 20]. Railsback [26] emphasizes the need for re-implementing models in diverse programming languages to capture and represent the complexity of real-world systems. Edmond and Hales [14] state that replication can reveal surprising flaws, even in the simplest of models. Chattoe-Brown et al. [12] emphasize that ensuring such replication becomes even more critical when the model outcomes have the potential to impact individuals’ lives significantly.

Unfortunately, replication of ABMs is underused in practice. Zhong and Kim [33] elaborate on possible challenges that explain why re-implementation is still rare. They emphasize that replication is a highly resource-demanding activity with relatively low payoffs in the form of publishable articles, sometimes seen as a trivial activity given to students who take their first steps in coding.

This article attempts to aid in building procedures that support replication [27, 30, 32], recognizing the importance of the original research process that starts with the conceptual model. The aim is to report on a systematic process of model replication, sharing good practices and lessons learned from re-implementing the HUMAT socio-cognitive architecture in Python (following an original implementation in NetLogo). The following section introduces a systematic guide for re-implementing agent-based models - a step-by-step process of model re-implementation. We developed this guide alongside the re-implementing HUMAT in Python case study. Effort was taken to generalize the re-implementation process. The guide proposed here serves as a starting point, aimed to be further developed. The article concludes with a short discussion.

2 Guide for Re-implementing Agent-Based Models

Re-implementing an existing agent-based model in a different programming language involves a series of steps to ensure the new implementation is accurate, efficient, and maintainable. We propose the following systematic approach to guide the re-implementation process (Figure 1), summarized in the most important steps below.

Understand the original model: Before beginning any re-implementation, it is essential to clearly understand the existing model’s functionality and designs [24]. This allows the developer to identify potential issues or limitations that should be addressed in the new implementation. Hence, we need to start by studying the original model’s documentation, code, and any related publications and gain a thorough understanding of its objectives, assumptions, agents, behaviors, interactions, environment, and other relevant aspects.

Refer to caption — Figure 1: The process for re-implementing ABM Models

Design a generic model: If the original model’s documentation is tightly coupled with the original programming language, we must outline a generic model independent of a programming language and framework. The generic model should describe the objective, assumptions, agents, behaviors, interactions, and environment. In that sense, applying UML and object-oriented patterns [19] and pattern-oriented agent modeling [15] are good candidates.

Choose a new programming language¹¹1Note that the initiation of this step is independent from initiation of the other steps and can start at any time.: Choosing the correct programming language can significantly impact its success and depends on several factors [25, 23]. The criteria to be considered are the target platform, target users, (if any) partners’ experience/preference, and the language’s community, library, and support strength. Common choices include Python, Java, and NetLogo [1].

Identify appropriate libraries or frameworks: Research and choose libraries or frameworks that are compatible with your chosen programming language and can facilitate agent-based modeling. For example, Mesa for Python [21], Repast for Java [13], or NetLogo’s built-in constructs/extensions.

Design the new model: Based on the generic model and considering the chosen language and framework, design a new model representing agents, environments, interactions, and behaviors. Consider whether any modifications, adaption of the data structures, or optimizations should be made to the generic model based on the new programming language’s capabilities.

Implement the new model: Translate the design model into the chosen programming language, adapting the structure and syntax as needed. Use the chosen libraries or frameworks to help streamline the process.

Validate the new model: Test the new model against the original to ensure it produces the same or similar results [18, 17]. This may involve comparing outputs, such as agent behaviors, interactions, aggregate patterns, and any performance metrics. Address any discrepancies or issues that arise.

Document the new model: Create thorough documentation for the new model, including explanations of its purpose, assumptions, agents, behaviors, interactions, and environment. In that sense, the ODD protocol [16] or UML-based specifications [19] can be used. Include information on any changes or optimizations made during the re-implementation process.

Share and collaborate: Share the new model with the original model’s authors and the broader research community through platforms like CoMSES²²2CoMSES Model Library, https://www.comses.net/codebases/, last access on 11/05/2023., GitLab, GitHub, and through scientific journals and conferences. Solicit feedback, collaborate on improvements, and contribute to the growing body of knowledge in agent-based modeling.

3 Case Study: Re-Implementing HUMAT

We have chosen a realistic case study to validate the effectiveness of the proposed re-implementation process. In the following, we present how we followed the abovementioned guideline (Section 2) in three subsections.

3.1 Choosing the Programming Language and Identifying the Libraries/Frameworks

In our case, the need for re-implementation was driven by the goal of the URBANE project³³3URBANE, https://www.urbane-horizoneurope.eu, last access on 10/05/2023. that requires combining the elements of two different simulation models: HUMAT [5] (implemented in NetLogo) and MASS-GT [11] (implemented in Python) into a single simulation model. Since the target of the resulting model will be used by our partner who knows Python and integrating HUMAT will be easier if we have a Python version, we decided to re-implement HUMAT in Python.

NetLogo is a well-documented ABM platform that uses a primary object-oriented language with primitives (predefined keywords) to control agents. Python is a general-purpose, high-level programming language. For the URBANE implementation, we used the Mesa ABM framework [21]. Mesa extends Python with several functionalities to make programming ABMs more manageable. While it is less comprehensive and well-documented than NetLogo, it offers modelers the benefit of accessing many Python libraries.

3.2 Understanding HUMAT and Designing its Generic Model

To understand HUMAT, we used the available documents and publications [5, 6, 7, 8], and its corresponding NetLogo version (Figure 2).

As a result of the understanding process, the purpose of the HUMAT model is to represent agents’ socio-cognitive process of attitude formation. The subjects of the attitude – the options an agent decides between (alternative A and alternative B) are decided by the modeler to fit the research problem that the agent-based model investigates.

The model is composed mainly of HUMAT agents connected through one or several social networks (i.e., ego networks). Each HUMAT agent is characterized by a set of needs/motives that are important for the subject of the attitude that can belong to one of three groups: experiential needs, social needs, and values. HUMAT agents vary regarding the importance of each motive and how the choice alternatives satisfy each motive. When HUMAT agents form their attitude toward a choice alternative, they reflect on how satisfying that alternative is. If the alternative satisfies one motive and dissatisfies another motive (i.e., has pros and cons), a HUMAT agent experiences an unpleasant state of dissonance. Consequently, that agent faces a dilemma and employs one of two dissonance resolution strategies to maintain cognitive consistency. Suppose the dilemma is non-social (i.e., the social need to be surrounded by enough like-minded HUMATS is satisfied). In that case, the HUMAT inquires - strives to change its own beliefs by asking the most persuasive alter in the ego network for advice. If the dilemma is social (i.e., the social need is dissatisfied), the HUMAT signals to the most gullible alter, striving to persuade them to change their mind.

To do this effectively, each HUMAT has a representation of all alters linked to it in the ego network. An activated link between HUMAT and the targeted alter denotes a communication act - sharing information about the subject of the attitude (either inquiring or signaling). The persuasiveness of the communicating agent depends on similarity and aspirational characteristics relevant to a given research context.

Based on the above understanding, we designed a programming language-independent generic model for HUMAT (Figure 3 and Figure 4). Figure 3 depicts the high-level representations of various concepts in the HUMAT domain and their relationships. Figure 4 represents an overall behavioral model for a HUMAT agent within a social network. The model initializes nodes (HUMATS) and edges in the social network, creating agent instances, and initializing their variables, motives, and choices. Then, it adds the agents to the network, initializes their representations of other agents (alters), and updates their social motives for choices. During each simulation step (tick), agents may decide to signal, inquire, or do nothing based on their current dilemmas and the dissonance strength of their chosen action. If an agent is not satisfied with their choice, they will try to become more content by signalling or inquiring. The basic version of the HUMAT architecture assumes perfect information about alter choices, meaning that all choices are visible to other agents in the ego network. Throughout the simulation, the agents continuously update their alter representations, social motives of choices and make new choices based on their evaluations of motives and dissonance strength.

3.3 Reimplementing HUMAT in Python

Re-implementing HUMAT in Python from the generic conceptual model is a straightforward process. Each concept in the generic model is translated into a Python class with related parameters and methods. The two main classes of the model describe the agents (HumatAgent) and the model (HumatModel) (see Figure 5).

The HumatModel class extends the Mesa Model class and controls methods executed during a time step. The HumatAgent class extends the Mesa Agent class and controls the methods executed by the agent. The generic model does not specify which Python data types, syntax, and packages to use. These decisions are up to the modeler and depend on their personal experiences.

3.4 Validating HUMAT in Python

We start each validation by configuring both models identically by importing all model states of the NetLogo model into Python after initial initialization. Subsequently, automatic unit tests of agent parameters are executed at each time step. This process is repeated, considering increasing agent populations and degrees of randomization (e.g., by controlling scheduling). Throughout the testing, the methods’ functionality is reviewed and optimized. The findings of this comprehensive case study will be documented in a separate paper.

4 Discussion

Replicating a NetLogo model directly in Python poses some specific challenges in the implementation process. A brief description of the main challenges we faced is given below. More complete comparisons between NetLogo and Python can be found in [2].

•

Object-oriented coding and methods: The NetLogo model is written as a collection of procedures: (i) a setup defines agent parameters and the model environment (e.g., patches and networks), (ii) the main go loop is then executed to run all model procedures for a defined number of time steps (ticks), (iii) the remaining procedures are non-restrictive and written anywhere below the setup and go procedure. The Python model is organized into classes with specific methods: (i) the main class contains methods defining the model inputs and the number of time steps, (ii) the model class controls methods executed during a time step, (iii) the §agent class control the methods that agents execute. In NetLogo, two types of procedures are: to and to-report. The to procedures usually contain a set of commands executed (e.g., by agents), while the to-report returns a value. Functions and methods in Python can execute commands or return values.
•

Turtles and Breed vs. Agent classes: NetLogo has four predefined agent types: turtles, patches, links, and the observer. Breeds are used to define specific sub-groups of agents (e.g., HUMATS). Each agent and breed can have specific parameters assigned to it and can be controlled using NetLogo keywords, the primitives. Python, on the other hand, uses classes to define objects. One of the features of the Mesa is the Agent class. Each object created as a sub-class of Agent is automatically equipped with a unique id and a step() method and inherits features.
•

Agentset, lists, and dictionaries: In NetLogo, groups of agents are organised in agentsets. These sets of agents can be created on the fly in random order. Agentsets are a very comfortable way to control or select a subset of agents using a set of primitives. While Python can create agent sets, storing agents in dictionaries is often more convenient.

Due to these challenges, it is not practical to re-implement a model in Python directly from a NetLogo model. The difference in abstractions used in both languages will make it hard for the modeler to transition. Consequently, for an effective re-implementation and rapid re-implementation in other programming languages, abstracting away the programming language concept and designing a generic model is essential. For instance, thanks to this generic model, we plan to re-implement HUMAT in Java for another project, and it will be pretty rapid.

5 Conclusions and Future Work

This paper contributes to the literature in three meaningful ways. One, previous studies agree on the importance of replicating agent-based models, however they mostly present experiences on individual models (e.g., [4, 29, 20]). Here, we add to the existing general guidelines [32] by proposing a programming language-independent systematic approach, from understanding the existing model to sharing the new implementation.

Two, replications of ABMs focus on discussing the validation of the re-implemented model: to what extent the outputs of the re-implemented model are aligned with the outputs of the original model [10]. The case study of replicating HUMAT described here focuses on the re-implementation process rather than the model outcomes.

Three, the authors provide a glimpse of the re-implementation process report having developed a general, conceptual model that is the basis of the original ABM [30] or a platform-independent model [27]. This is a similar approach to developing a generic model proposed here. An intermediate, generic model enables a focus on the investigated phenomenon without anchoring in the concepts present in a given programming language. Additionally, it makes further re-implementations in different languages faster and less effortful.

Up until now, we closely followed the guideline until the Validate the New Model step. This remaining step involves sensitivity analysis and testing of the new model and thus requires a more detailed discussion. In future work, we will finish validating the new model implemented in Python and report the results of our experience.

We hope that, in future, the guidelines will be used and perfected by the social simulation community. To make re-implementation of ABMs more common, modellers should follow the Share and collaborate step of the proposed guideline. The social simulation community can popularize such works by initiating a dedicated label in COMSES or launching a publication outlet focusing on model evaluation, replication and re-implementation.

5.0.1 Acknowledgements

The work reported here is part of the URBANE project, which has received funding from the European Union’s Horizon Europe Innovation Action under grant agreement No. 101069782. We thank the reviewers for the thoughtful remarks, especially related to the popularization ideas.

References

[1] Abar, S., Theodoropoulos, G.K., Lemarinier, P., O’Hare, G.M.P.: Agent Based Modelling and Simulation tools: A review of the state-of-art software. Computer Science Review 24, 13–33 (2017)
[2] Abbott, R., Lim, J.: PyLogo: A Python Reimplementation of (Much of) NetLogo:. In: Proceedings of the 11th International Conference on Simulation and Modeling Methodologies, Technologies and Applications. pp. 199–206. SCITEPRESS - Science and Technology Publications, Online Streaming (2021)
[3] Achter, S., Borit, M., Chattoe-Brown, E., Siebers, P.O.: RAT-RS: a reporting standard for improving the documentation of data use in agent-based modelling. International Journal of Social Research Methodology 25(4), 517–540 (Jul 2022)
[4] An, G., Mi, Q., Dutta-Moscato, J., Vodovotz, Y.: Agent-based models in translational systems biology. Wiley interdisciplinary reviews. Systems biology and medicine 1(2), 159–171 (2009)
[5] Antosz, P., Jager, W., Polhill, J.G., Salt, D., Alonso-Betanzos, A., Sánchez-Maroño, N., Guijarro-Berdiñas, B., Rodríguez, A.: Simulation model implementing different relevant layers of social innovation, human choice behaviour and habitual structures. Tech. Rep. D7.2 (2019)
[6] Antosz, P., Jager, W., Polhill, J.G., Salt, D., Alonso-Betanzos, A., Sánchez-Maroño, N., Guijarro-Berdiñas, B., Rodríguez, A., Scalco, A.: SMARTEES simulation implementations. Tech. Rep. D7.3 (2021)
[7] Antosz, P., Puga-Gonzalez, I., Shults, F.L., Lane, J.E., Normann, R.: Documenting Data Use in a Model of Pandemic “Emotional Contagion” Using the Rigour and Transparency Reporting Standard (RAT-RS). In: Czupryna, M., Kamiński, B. (eds.) Advances in Social Simulation, pp. 439–451. Springer, Cham (2022)
[8] Antosz, P., Puga-Gonzalez, I., Shults, F.L., Szczepanska, T.: HUM-e: An emotive-socio-cognitive agent architecture for representing human decision-making in anxiogenic contexts. In: Squazzoni, F. (ed.) Advances in Social Simulation. Springer International Publishing, Cham
[9] Antosz, P., Szczepanska, T., Bouman, L., Polhill, J.G., Jager, W.: Sensemaking of causality in agent-based models. International Journal of Social Research Methodology 25(4), 557–567 (Jul 2022). https://doi.org/10.1080/13645579.2022.2049510
[10] Axelrod, R.: Advancing the Art of Simulation in the Social Sciences. In: Simulating Social Phenomena, Lecture Notes in Economics and Mathematical Systems, vol. 456, pp. 21–40. Springer Berlin Heidelberg, Berlin, Heidelberg (1997)
[11] de Bok, M., Tavasszy, L.: An empirical agent-based simulation system for urban goods transport (MASS-GT). Procedia Computer Science 130, 126–133 (2018)
[12] Chattoe-Brown, E., Gilbert, N., Robertson, D.A., Watts, C.: Reproduction as a Means of Evaluating Policy Models: A Case Study of a COVID-19 Simulation. medRxiv (2021). https://doi.org/10.1101/2021.01.29.21250743
[13] Collier, N.: RePast: An Extensible Framework for Agent Simulation. The University of Chicago’s Social Science Research (2003)
[14] Edmonds, B., Hales, D.: Replication, Replication and Replication: Some Hard Lessons from Model Alignment (Oct 2003)
[15] Grimm, V., Railsback, S.F.: Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology. Philosophical Transactions of the Royal Society B: Biological Sciences 367(1586), 298–310 (2012)
[16] Grimm, V., Railsback, S.F., Vincenot, C.E., Berger, U., Gallagher, C., DeAngelis, D.L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J., Johnston, A.S.A., Milles, A., Nabe-Nielsen, J., Polhill, J.G., Radchuk, V., Rohwäder, M.S., Stillman, R.A., Thiele, J.C., Ayllón, D.: The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism. Journal of Artificial Societies and Social Simulation 23(2), 7 (2020)
[17] Gürcan, O., Dikenelli, O., Bernon, C.: Towards a Generic Testing Framework for Agent-Based Simulation Models. In: Ganzha, M., Maciaszek, L.A., Paprzycki, M. (eds.) FedCSIS 2011. pp. 635–642. Szczecin, Poland (Sep 2011)
[18] Gürcan, O., Dikenelli, O., Bernon, C.: A generic testing framework for agent-based simulation models. In: Agent-Based Modeling and Simulation, pp. 231–270. Springer (2014)
[19] Larman, C.: Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development (3rd Edition). Prentice Hall, USA (2004)
[20] Liang, H., Fu, K.w.: Testing Propositions Derived from Twitter Studies: Generalization and Replication in Computational Social Science. PLOS ONE 10(8), 1–14 (Aug 2015). https://doi.org/10.1371/journal.pone.0134270
[21] Masad, D., Kazil, J.: Mesa: An Agent-Based Modeling Framework. pp. 51–58. Austin, Texas (2015). https://doi.org/10.25080/Majora-7b98e3ed-009
[22] Maxwell, S.E., Lau, M.Y., Howard, G.S.: Is psychology suffering from a replication crisis? Am Psychol. 70(6), 487–498 (2015)
[23] North, M.J., Macal, C.M.: Agent Based Modeling and Computer Languages. In: Meyers, R.A. (ed.) Encyclopedia of Complexity and Systems Science, pp. 131–148. Springer New York (2009). https://doi.org/10.1007/978-0-387-30440-3_8
[24] Pressman, R., Maxim, B.: Software Engineering: A Practitioner’s Approach, 8th Ed (Jan 2014)
[25] Railsback, S., Grimm, V.: Agent-Based and Individual-Based Modeling: A Practical Introduction. Agent-based and Individual-based Modeling: A Practical Introduction, Princeton University Press (2019)
[26] Railsback, S.F.: Concepts from complex adaptive systems as a framework for individual-based modelling. Ecological Modelling 139(1), 47–62 (2001)
[27] Sansores, C., Pavón, J.: Agent-Based Simulation Replication: A Model Driven Architecture Approach. In: MICAI 2005: Advances in AI 4th Mexican Int. Conf. on AI, LNAI, vol. 3789, pp. 244–253. Springer (2005)
[28] Tang, W., Grimm, V., Tesfatsion, L., Shook, E., Bennett, D., An, L., Gong, Z., Ye, X.: Code Reusability and Transparency of Agent-Based Modeling: A Review from a Cyberinfrastructure Perspective. In: Tang, W., Wang, S. (eds.) High Performance Computing for Geospatial Applications, pp. 115–134. Springer, Cham (2020)
[29] Thiele, J.C., Kurth, W., Grimm, V.: Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and ’R’. Journal of Artificial Societies and Social Simulation 17(3), 11 (2014)
[30] Wilensky, U., Rand, W.: Making Models Match: Replicating an Agent-Based Model. Journal of Artificial Societies and Social Simulation 10(4) (2007)
[31] Will, O., Hegselmann, R.: A Replication That Failed: On the Computational Model in ’Michael W. Macy and Yoshimichi Sato: Trust, Cooperation and Market Formation in the U.S. and Japan. JASSS 11(3) (2008)
[32] Zhang, J., Robinson, D.T.: Replication of an agent-based model using the Replication Standard. Environmental Modelling & Software 139, 105016 (2021)
[33] Zhong, W., Kim, Y.: Using Model Replication to Improve Reliability of Agent-Based Models. In: Chai, S.K., Salerno, J.J., Mabry, P.L., Hutchison, D., Kanade, T. (eds.) Advances in Social Computing: 3rd Int. Conf. on Social Computing, Behavioral Modeling, and Prediction, LNCS, vol. 6007, pp. 118–127. Springer (2010)