Tag Archives: MarijnKeijzer

Content

Make some noise! Why agent-based modelers should embrace the power of randomness

Leave a comment

By Peter Steiglechner1, Marijn Keijzer2

1 Complexity Science Hub, Austria; steiglechner@csh.ac.at
2 Institute for Advanced Study in Toulouse, France

Abstract

‘Noisy’ behavior, belief updating, or decision-making is universally observed, yet typically treated superficially or not even accounted for at all by social simulation modelers. Here, we show how noise can affect model dynamics and outcomes, argue why injecting noise should become a central part of model analyses, and how it can help our understanding of (our mathematical models for) social behavior. We formulate some general lessons from the literature around noise, and illustrate how considering a more complete taxonomy of different types of noise may lead to novel insights.

‘Flooding the zone’ with noise

In his inaugural address in January 2025, US president Trump announced that he would “tariff and tax foreign countries to enrich [US] citizens”. Since then, Trump has flooded the world news with a back-and-forth of threatening, announcing, and introducing tariffs, only to pause, halt, or even revoke them within a matter of days. Trump’s statements on tariffs are just one (albeit rather extreme) example of how noisy and ambiguous political signaling can be. Ambiguity in politics can be strategic (Page, 1976), but it can also simply result from a failure to accurately describe one’s position. Most of us are probably familiar with examples of noise in our own personal lives as well—we may wholeheartedly support one thing, and take a skeptical stance in the next discussion. People have always been inherently noisy (Vul & Pashler, 2008; Kahneman, Sibony, & Sunstein, 2021). But the pervasiveness of noise has become particularly evident in recent years, as social media have made it easier to frequently signal our political opinions (e.g. through like-buttons) and to track the noisy or inconsistent behaviors of others.

Noise can flip model dynamics

As quantitative scientists, most of us are not aware of how important noise can be. Conventional statistical models used in the social sciences typically assume noise away. This is because unexplained variance in simple regression models—if not too abnormally distributed—should not affect the validity of the results; so why should we care? Social simulation models play by different rules. With strictly operating behavioral rules on the micro-level and strong interdependence, noise on the individual level plays a pivotal role in shaping collective outcomes. The importance of noise contrasts with the fact that many models still assume that individual-level properties and actions are fully deterministic, consistent, accurate, and certain.

For example, opinion dynamics models like the Bounded Confidence model (Deffuant et al., 2000; Hegselmann & Krause, 2002) and the Dissemination of Culture model (Axelrod, 1997), both illustrate how global diversity (or ‘polarization’) emerges because of local homogeneity (or ‘consensus’). But this core principle is highly dependent on the absence of noise! The persistence of different cultural areas completely collapses under even the smallest probability of differentiation (Klemm et al., 2003; Flache & Macy, 2011), and fragmentation and polarization become unlikely when agents sometimes independently change their opinions (Pineda, Toral, & Hernández-García, 2011). Similarly, adding noise to the topology of connections can drastically change the dynamics of diffusion and contagion (Centola & Macy, 2007). In computational, agent-based models of social systems, noise does not necessarily cancel out. Many social processes are complex, path-dependent, computationally irreducible and highly non-linear. As such, noise can trigger cascades of ‘errors’ that lead to statistically and qualitatively different behaviors (Macy & Tsvetkova, 2015).

What is noise? What is it not?

There is no one way to introduce noise, or to dedicate and define a source of noise. Noise comes in different shapes and forms. When introducing noise into a model of social phenomena, there are some important lessons to consider:

  1. Noise is not just unpredictable randomness. Instead, noise often represents uncertainty (Macy & Tsvetkova, 2015), which can mean a lack of precision in measurements, ambiguity, or inconsistency. Heteroskedasticity—or the fact that the variance of noise depends on the context—is more than a nuisance in statistical estimation. In ABM research in particular, the variance of uncertainty can be a source of nonlinearity. As such, introducing noise into a model should not be equated merely with ‘running the simulations with different random seeds’ or ‘drawing agent attributes from a normal distribution’.
  2. Noise enters social systems in many ways and in every aspect of the system. This includes noisy observations of others or the environment (Gigerenzer & Brighton, 2009), noisy transmission of signals (McMahan & Evans, 2018), noisy application of heuristics (Mäs & Nax, 2016), noisy interaction patterns (Lloyd-Smith et al., 2005), heterogeneity across societies and across individuals (Kahneman, Sunstein, & Sibony, 2021), and inconsistencies over time (Vul & Pashler, 2008). This is crucial because noise representing different forms and different sources of uncertainty or randomness can affect social phenomena such as social influence, consensus, and polarization in quite distinct ways (as we will outline in the next section).
  3. Noise can be adaptive and heterogeneous across individuals. Noise is not a passive property of a system, but can be a context-dependent, dynamically adapted strategy (Frankenhuis, Panchanathan, & Smaldino, 2022). For example, some individuals tend to be more covert and less precise than others for instance when they perceive themselves to be in a minority (Smaldino & Turner, 2022). Some situations require individuals to be more noisy or unpredictable, like when taking a penalty in soccer, other situations less so, such as writing an online dating ad. People need to decide and adapt the degree of noise in their social signals. Smaldino et al. (2023) highlighted that all strategies that lead collectives to perform well in solving complex tasks depend in some way on maintaining (but also adapting to) a certain level of transient noise.
  4. Noise is itself a signal. There are famous examples of institutions or individuals using noise signaling to spread doubt and uncertainty in debates about climate change or the health effects of smoking (see ‘Merchants of Doubt’ by Oreskes & Conway, 2010). Such actors signal noise to diffuse and discredit valuable information. One could certainly argue that Trump’s noisy stance on tariffs also falls into this category. 

In short, noise represents meaningful, multi-faceted, adaptive, and strategic aspects of a system. In social systems—which are, by definition, systems of interdependence—noise is essential to understanding that system. As Macy & Tsvetkova put it: ‘strip away the noise and you may strip away the explanation’ (2015).

A taxonomy of noise in opinion dynamics

In our paper ‘Noise and opinion dynamics’ published last year in Royal Society of Open Science, we reviewed and examined if and how different sources of noise affect the results in a model of opinion dynamics (Steiglechner et al., 2024). The model builds on the bounded confidence model by Deffuant et al. (2000), calibrated on a survey measuring environmental attitudes. We identified at least four different types of noise in a system of opinion formation through dyadic social influence: exogenous noise, selectivity noise, adaptation noise and ambiguity noise.

Figure 1. Sources of noise in social influence models for opinion dynamics (adapted from Steiglechner et al., 2024)

Each type of noise in our taxonomy enters at a different stage of the interaction process (as shown in Figure 1). Ambiguity and adaptation noise both depend on the current attitudes of the sender and the receiver, respectively, whereas selectivity noise acts on the connections between individuals. Exogenous noise is a ‘catch-all’ category of noise added to the agent’s attributes regardless of the (success of) interaction. Some of these types of noise may have similar effects on population-level opinion dynamics in the thermodynamic limit (Nugent , Gomes, & Wolfram; 2024). But they can lead to quite different trajectories and conclusions about the noise effect when we look at real cases of finite-size and finite-time simulations.

Previous work had established that even small amounts of noise can affect the tendency of the bounded confidence model to produce complete consensus or multiple fragmented, internally coherent groups, but our results highlight that different types of noise can have quite distinct signatures. For example, while selectivity noise always increases the coherence of groups, only intermediate levels of exogenous noise unite individuals. Moreover, exogenous noise leads to convergence because it destroys the internal coherence within the fragmented groups, whereas selectivity noise leads to convergence because it connects polarized individuals across these groups. Ambiguity noise has yet another signature. For example, while low levels of ambiguity have no effect on fragmentation (similar to exogenous and adaptation noise), intermediate and even high levels of ambiguity can produce a somewhat coherent majority-group (similar to selectivity noise). More importantly, ambiguity noise also produces drift: a gradual shift in the average opinion toward a more extreme position (Steiglechner et al., 2024). This is a remarkable result, because not only does ambiguous messaging alter the robustness of the clean, noiseless model, it actually produces a novel type of extremization using only positive influence!

Make some noise!

The above taxonomy is, of course, only a starting point for further discussion: It is not comprehensive and does not take into account adaptiveness or strategy. However, already this variety of effects of the different types of noise on consensus, polarization, and social influence should make us more aware of noise in general—not just as an ‘afterthought’ or a robustness check, but as a modeling choice that represents a critical component of the model. Many modeling studies do consider how noise can affect the model outputs, but it matters—a lot—where and how they introduce noise (see also De Sanctis & Galla, 2009).

Noise is an essential aspect of human behavior, social systems, and politics, as Trump’s back-and-forth on tariffs illustrates quite effectively these days. When studying social phenomena such as opinion formation and polarization, we should take the effects of noise as seriously as the effects of behavioral biases or heuristics (Kahneman, Sunstein, & Sibony, 2021). That is, while we social systems modelers tend to spend a lot of time to formulate, justify, and analyze behavioral rules of individuals—generally considered the core of the model—, we should devote more time to formalize what kind of noise enters the modeled system where and how and analyze how this affects the dynamics (as also argued in the exchange of letters between Kahneman et al. and Krakauer & Wolpert, 2022). Noise is a meaningful, multi-faceted, adaptive, and strategic component of social systems. Rather than ‘just a robustness check’, it is a fundamental ingredient of the modeled system—a type of behavior in itself—and, thus, an object of study on its own. This is a call to all modelers (in the house) to make some noise!

Acknowledgments

We thank Victor Møller Poulsen and Paul E. Smaldino for their feedback.

References

Axelrod, R. (1997). The dissemination of culture: A model with local convergence and global polarization. Journal of Conflict Resolution, 41(2), 203–226. https://doi.org/10.1177/0022002797041002001

Centola, D., & Macy, M. (2007). Complex contagions and the weakness of long ties. American Journal of Sociology, 113(3), 702–734. https://doi.org/10.1086/521848

Deffuant, G., Neau, D., Amblard, F., & Weisbuch, G. (2000). Mixing beliefs among interacting agents. Advances in Complex Systems, 3(1–4), 87–98. https://doi.org/10.1142/S0219525900000078

De Sanctis, L., & Galla, T. (2009). Effects of noise and confidence thresholds in nominal and metric Axelrod dynamics of social influence. Physical Review E, 79(4), 046108. https://doi.org/10.1103/PhysRevE.79.046108

Flache, A., & Macy, M. W. (2011). Local convergence and global diversity: From interpersonal to social influence. Journal of Conflict Resolution, 55(6), 970–995. https://doi.org/10.1177/0022002711414371

Frankenhuis, W. E., Panchanathan, K., & Smaldino, P. E. (2023). Strategic ambiguity in the social sciences. Social Psychological Bulletin, 18, e9923. https://doi.org/10.32872/spb.9923

Gigerenzer, G., & Brighton, H. (2009). Homo heuristicus: Why biased minds make better inferences. Topics in Cognitive Science, 1(1), 107–143. https://doi.org/10.1111/j.1756-8765.2008.01006.x

Hegselmann, R., & Krause, U. (2002). Opinion dynamics and bounded confidence models, analysis and simulation. Journal of Artificial Societies and Social Simulation, 5(3). http://jasss.soc.surrey.ac.uk/5/3/2.html

Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A flaw in human judgment. New York: Little, Brown Spark.

Kahneman, D., Krakauer, D.C., Sibony, O., Sunstein, C. and Wolpert, D. (2022) ‘An exchange of letters on the role of noise in collective intelligence’, Collective Intelligence, 1(1), p. 26339137221078593. doi: https://doi.org/10.1177/26339137221078593.

Klemm, K., Eguíluz, V. M., Toral, R., & Miguel, M. S. (2003). Global culture: A noise-induced transition in finite systems. Physical Review E, 67(4), 045101. https://doi.org/10.1103/PhysRevE.67.045101

Lloyd-Smith, J. O., Schreiber, S. J., Kopp, P. E., & Getz, W. M. (2005). Superspreading and the effect of individual variation on disease emergence. Nature, 438(7066), 355–359. https://doi.org/10.1038/nature04153

Macy, M., & Tsvetkova, M. (2015). The signal importance of noise. Sociological Methods & Research, 44(2), 306–328. https://doi.org/10.1177/0049124113508093

Mäs, M., & Nax, H. H. (2016). A behavioral study of “noise” in coordination games. Journal of Economic Theory, 162, 195–208. https://doi.org/10.1016/j.jet.2015.12.010

McMahan, P., & Evans, J. (2018). Ambiguity and engagement. American Journal of Sociology, 124(3), 860–912. https://doi.org/10.1086/701298

Nugent, A., Gomes, S. N., & Wolfram, M.-T. (2024). Bridging the gap between agent based models and continuous opinion dynamics. Physica A: Statistical Mechanics and its Applications, 651, 129886. https://doi.org/10.1016/j.physa.2024.129886

Oreskes, N., & Conway, E. M. (2010). Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. New York: Bloomsbury Press.

Page, B. I. (1976). The theory of political ambiguity. American Political Science Review, 70(3), 742–752. https://doi.org/10.2307/1959865

Pineda, M., Toral, R. & Hernández-García, E. (2011) ‘Diffusing opinions in bounded confidence processes’, The European Physical Journal D, 62(1), pp. 109–117. doi: 10.1140/epjd/e2010-00227-0.

Smaldino, P. E., & Turner, M. A. (2022). Covert signaling is an adaptive communication strategy in diverse populations. Psychological Review, 129(4), 812–829. https://doi.org/10.1037/rev0000344

Smaldino, P. E., Moser, C., Pérez Velilla, A., & Werling, M. (2023). Maintaining transient diversity is a general principle for improving collective problem solving. Perspectives on Psychological Science, Advance online publication. https://doi.org/10.1177/17456916231180100

Steiglechner, P., Keijzer, M. A., Smaldino, P. E., Moser, D., & Merico, A. (2024). Noise and opinion dynamics: How ambiguity promotes pro-majority consensus in the presence of confirmation bias. Royal Society Open Science, 11(4), 231071. https://doi.org/10.1098/rsos.231071

Vul, E., & Pashler, H. (2008). Measuring the crowd within: Probabilistic representations within individuals. Psychological Science, 19(7), 645–647. https://doi.org/10.1111/j.1467-9280.2008.02136.x

Steiglechner, P. & Keijzer, M.(2025) Make some noise! Why agent-based modelers should embrace the power of randomness. Review of Artificial Societies and Social Simulation, 30 May 2025. https://rofasss.org/2025/05/31/noise

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

If you want to be cited, calibrate your agent-based model: A Reply to Chattoe-Brown

1 Comment

By Marijn A. Keijzer

This is a reply to a previous comment, (Chattoe-Brown 2022).

The social simulation literature has called on its proponents to enhance the quality and realism of their contributions through systematic validation and calibration (Flache et al., 2017). Model validation typically refers to assessments of how well the predictions of their agent-based models (ABMs) map onto empirically observed patterns or relationships. Calibration, on the other hand, is the process of enhancing the realism of the model by parametrizing it based on empirical data (Boero & Squazzoni, 2005). We would expect that presenting a validated or calibrated model serves as a signal of model quality, and would thus be a desirable characteristic of a paper describing an ABM.

In a recent contribution to RofASSS, Edmund Chattoe-Brown provocatively argued that model validation does not bear fruit for researchers interested in boosting their citations. In a sample of articles from JASSS published on opinion dynamics he observed that “the sample clearly divides into non-validated research with more citations and validated research with fewer” (Chattoe-Brown, 2022). Well-aware of the bias and limitations of the sample at hand, Chattoe-Brown calls on refutation of his hypothesis. An analysis of the corpus of articles in Web of Science, presented here, could serve that goal.

To test whether there exists an effect of model calibration and/or validation on the citation counts of papers, I compare citation counts of a larger number of original research articles on agent-based models published in the literature. I extracted 11,807 entries from Web of Science by searching for items that contained the phrases “agent-based model”, “agent-based simulation” or “agent-based computational model” in its abstract.[1] I then labeled all items that mention “validate” in its abstract as validated ABMs and those that mention “calibrate” as calibrated ABMs. This measure if rather crude, of course, as descriptions containing phrases like “we calibrated our model” or “others should calibrate our model” are both labeled as calibrated models. However, if mentioning that future research should calibrate or validate the model is not related to citations counts (which I would argue it indeed is not), then this inaccuracy does not introduce bias.

The shares of entries that mention calibration or validation are somewhat small. Overall, just 5.62% of entries mention validation, 3.21% report a calibrated model and 0.65% fall in both categories. The large sample size, however, will still enable the execution of proper statistical analysis and hypothesis testing.

How are mentions of calibration and validation in the abstract related to citation counts at face value? Bivariate analyses show only minor differences, as revealed in Figure 1. In fact, the distribution of citations for validated and non-validated ABMs (panel A) is remarkably similar. Wilcoxon tests with continuity correction—the nonparametric version of the simple t test—corroborate their similarity (W = 3,749,512, p = 0.555). The differences in citations between calibrated and non-calibrated models appear, albeit still small, more pronounced. Calibrated ABMs are cited slightly more often (panel B), as also supported by a bivariate test (W = 1,910,772, p < 0.001).

Picture 1

Figure 1. Distributions of number of citations of all the entries in the dataset for validated (panel A) and calibrated (panel B) ABMs and their averages with standard errors over years (panels C and D)

Age of the paper might be a more important determinant of citation counts, as panels C and D of Figure 1 suggest. Clearly, the age of a paper should be important here, because older papers have had much more opportunity to get cited. In particular, papers younger than 10 years seem to not have matured enough for its citation rates to catch up to older articles. When comparing the citation counts of purely theoretical models with calibrated and validated versions, this covariate should not be missed, because the latter two are typically much younger. In other words, the positive relationship between model calibration/validation and citation counts could be hidden in the bivariate analysis, as model calibration and validation are recent trends in ABM research.

I run a Poisson regression on the number of citations as explained by whether they are validated and calibrated (simultaneously) and whether they are both. The age of the paper is taken into account, as well as the number of references that the paper uses itself (controlling for reciprocity and literature embeddedness, one might say). Finally, the fields in which the papers have been published, as registered by Web of Science, have been added to account for potential differences between fields that explains both citation counts and conventions about model calibration and validation.

Table 1 presents the results from the four models with just the main effects of validation and calibration (model 1), the interaction of validation and calibration (model 2) and the full model with control variables (model 3).

Table 1. Poisson regression on the number of citations

# Citations
(1) (2) (3)
Validated -0.217*** -0.298*** -0.094***
(0.012) (0.014) (0.014)
Calibrated 0.171*** 0.064*** 0.076***
(0.014) (0.016) (0.016)
Validated x Calibrated 0.575*** 0.244***
(0.034) (0.034)
Age 0.154***
(0.0005)
Cited references 0.013***
(0.0001)
Field included No No Yes
Constant 2.553*** 2.556*** 0.337**
(0.003) (0.003) (0.164)
Observations 11,807 11,807 11,807
AIC 451,560 451,291 301,639
Note: *p<0.1; **p<0.05; ***p<0.01

The results from the analyses clearly suggest a negative effect of model validation and a positive effect of model calibration on the likelihood of being cited. The hypothesis that was so “badly in need of refutation” (Chattoe-Brown, 2022) will remain unrefuted for now. The effect does turn positive, however, when the abstract makes mention of calibration as well. In both the controlled (model 3) and uncontrolled (model 2) analyses, combining the effects of validation and calibration yields a positive coefficient overall.[2]

The controls in model 3 substantially affect the estimates from the three main factors of interest, while remaining in expected directions themselves. The age of a paper indeed helps its citation count, and so does the number of papers the item cites itself. The fields, furthermore, take away from the main effects somewhat, too, but not to a problematic degree. In an additional analysis, I have looked at the relationship between the fields and whether they are more likely to publish calibrated or validated models and found no substantial relationships. Citation counts will differ between fields, however. The papers in our sample are more often cited in, for example, hematology, emergency medicine and thermodynamics. The ABMs in the sample coming from toxicology, dermatology and religion are on the unlucky side of the equation, receiving less citations on average. Finally, I have also looked at papers published in JASSS specifically, due to the interest of Chattoe-Brown and the nature of this outlet. Surprisingly, the same analyses run on the subsample of these papers (N=376) showed a negative relationship between citation counts and model calibration/validation. Does the JASSS readership reveal its taste for artificial societies?

In sum, I find support for the hypothesis of Chattoe-Brown (2022) on the negative relationship between model validation and citations counts for papers presenting ABMs. If you want to be cited, you should not validate your ABM. Calibrated ABMs, on the other hand, are more likely to receive citations. What is more, ABMs that were both calibrated and validated are most the most successful papers in the sample. All conclusions were drawn considering (i.e. controlling for) the effects of age of the paper, the number of papers the paper cited itself, and (citation conventions in) the field in which it was published.

While the patterns explored in this and Chattoe-Brown’s recent contribution are interesting, or even puzzling, they should not distract from the goal of moving towards realistic agent-based simulations of social systems. In my opinion, models that combine rigorous theory with strong empirical foundations are instrumental to the creation of meaningful and purposeful agent-based models. Perhaps the results presented here should just be taken as another sign that citation counts are a weak signal of academic merit at best.

Data, code and supplementary analyses

All data and code used for this analysis, as well as the results from the supplementary analyses described in the text, are available here: https://osf.io/x9r7j/

Notes

[1] Note that the hyphen between “agent” and “based” does not affect the retrieved corpus. Both contributions that mention “agent based” and “agent-based” were retrieved.

[2] A small caveat to the analysis of the interaction effect is that the marginal improvement of model 2 upon model 1 is rather small (AIC difference of 269). This is likely (partially) due to the small number of papers that mention both calibration and validation (N=77).

Acknowledgements

Marijn Keijzer acknowledges IAST funding from the French National Research Agency (ANR) under the Investments for the Future (Investissements d’Avenir) program, grant ANR-17-EURE-0010.

References

Boero, R., & Squazzoni, F. (2005). Does empirical embeddedness matter? Methodological issues on agent-based models for analytical social science. Journal of Artificial Societies and Social Simulation, 8(4), 1–31. https://www.jasss.org/8/4/6.html

Chattoe-Brown, E. (2022) If You Want To Be Cited, Don’t Validate Your Agent-Based Model: A Tentative Hypothesis Badly In Need of Refutation. Review of Artificial Societies and Social Simulation, 1st Feb 2022. https://rofasss.org/2022/02/01/citing-od-models

Flache, A., Mäs, M., Feliciani, T., Chattoe-Brown, E., Deffuant, G., Huet, S., & Lorenz, J. (2017). Models of social influence: towards the next frontiers. Journal of Artificial Societies and Social Simulation, 20(4). https://doi.org/10.18564/jasss.3521

Keijzer, M. (2022) If you want to be cited, calibrate your agent-based model: Reply to Chattoe-Brown. Review of Artificial Societies and Social Simulation, 9th Mar 2022. https://rofasss.org/2022/03/09/Keijzer-reply-to-Chattoe-Brown

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

No one can predict the future: More than a semantic dispute

1 Comment

By Carlos A. de Matos Fernandes and Marijn A. Keijzer

(A contribution to the: JASSS-Covid19-Thread)

Models are pivotal to battle the current COVID-19 crisis. In their call to action, Squazzoni et al. (2020) convincingly put forward how social simulation researchers could and should respond in the short run by posing three challenges for the community among which is a COVID-19 prediction challenge. Although Squazzoni et al. (2020) stress the importance of transparent communication of model assumptions and conditions, we question the liberal use of the word ‘prediction’ for the outcomes of the broad arsenal of models used to mitigate the COVID-19 crisis by ours and other modelling communities. Four key arguments are provided that advocate using expectations derived from scenarios when explaining our models to a wider, possibly non-academic audience.

The current COVID-19 crisis necessitates that we implement life-changing policies that, to a large extent, build upon predictions from complex, quickly adapted, and sometimes poorly understood models. The examples of models spurring the news to produce catchphrase headlines are abundant (Imperial College, AceMod-Australian Census-based Epidemic Model, IndiaSIM, IHME, etc.). And even though most of these models will be useful to assess the comparative effectiveness of interventions in our aim to ‘flatten the curve’, the predictions that disseminate to news media are those of total cases or timing of the inflection point.

The current focus on predictive epidemiological and behavioural models brings back an important discussion about prediction in social systems. “[T]here is a lot of pressure for social scientists to predict” (Edmonds, Polhill & Hales, 2019), and we might add ‘especially nowadays’. But forecasting in human systems is often tricky (Hofman, Sharma & Watts, 2017). Approaches that take well-understood theories and simple mechanisms often fail to grasp the complexity of social systems, yet models that rely on complex supervised machine learning-like approaches may offer misleading levels of confidence (as was elegantly shown recently by Salganik et al., 2020). COVID-19 models appear to be no exception as a recent review concluded that “[…] their performance estimates are likely to be optimistic and misleading” (Wynants et al., 2020, p. 9). Squazzoni et al. describe these pitfalls too (2020: paragraph 3.3). In the crisis at hand, it may even be counter-productive to rely on complex models that combine well-understood mechanisms with many uncertain parameters (Elsenbroich & Badham, 2020).

Considering the level of confidence we can have about predictive models in general, we believe there is an issue with the way predictions are communicated by the community. Scientists often use ‘prediction’ to refer to some outcome of a (statistical) model where they ‘predict’ aspects of the data that are already known, but momentarily set aside. Edmonds et al. (2019: paragraph 2.4) state that “[b]y ‘prediction’, we mean the ability to reliably anticipate well-defined aspects of data that is not currently known to a useful degree of accuracy via computations using the model”. Predictive accuracy, in this case, can then be computed later on, by comparing the prediction to the truth. Scientists know that when talking about predictions of their models, they don’t claim to generalize to situations outside of the narrow scope of their study sample or their artificial society. We are not predicting the future, and wouldn’t claim we could. However, this is wildly different from how ‘prediction’ is commonly understood: As an estimation of some unknown thing in the future. Now that our models quickly disseminate to the general public, we need to be careful with the way we talk about their outcomes.

Predictions in the COVID-19 crisis will remain imperfect. In the current virus outbreak, society cannot afford to rely on the falsification of models for interventions against empirical data. As the virus remains to spread rapidly, our only option is to rely on models as a basis for policy, ceteris paribus. And it is precisely here – at ‘ceteris paribus’ – where the terminology ‘predictions’ miss the mark. All things will not be equal tomorrow, the next day, or the day after that (Van Bavel et al. [2020] note numerous topics that affect managing the COVID-19 pandemic and its impact on society). Policies around the globe are constantly being tweaked, and people’s behaviour changes dramatically as a consequence (Google, 2020). Relying on predictions too much may give a false sense of security.

We propose to avoid using the word ‘prediction’ too much and talk about scenarios or expectations instead where possible. We identify four reasons why you should avoid talking about prediction right now:

  1. Not everyone is acquainted with noise and emergence. Computational Social Scientists generally understand the effects of noise in social systems (Squazzoni et al., 2020: paragraph 1.8). Small behavioural irregularities can be reinforced in complex systems fostering unexpected outcomes. Yet, scientists not acquainted with studying complex social systems may be unfamiliar with the principles we have internalized by now, and put over-confidence in the median outputs of volatile models that enter the scientific sphere as predictions.
  2. Predictions do not convey uncertainty. The general public is usually unacquainted with academic esoteric concepts. For instance, showing a flatten-the-curve scenario generally builds upon mean or median approximation, oftentimes neglecting to include variability of different scenarios. Still, there are numerous other outcomes, building on different parameter values. We fear that by stating a prediction to an undisciplined public, they expect such a thing to occur for certain. If we forecast a sunny day, but there’s rain, people are upset. Talking about scenarios, expectations, and mechanisms may prevent confusion and opposition when the forecast does not occur.
  3. It’s a model, not a reality. The previous argument feeds into the third notion: Be honest about what you model. A model is a model. Even the most richly calibrated model is a model. That is not to say that such models are not informative (we reiterate: models are not a shot in the dark). Still, richly calibrated models based on poor data may be more misleading than less calibrated models (Elsenbroich & Badham, 2020). Empirically calibrated models may provide more confidence at face value, but it lies in the nature of complex systems that small measurement errors in the input data may lead to big deviations in outputs. Models present a scenario for our theoretical reasoning with a given set of parameter values. We can update a model with empirical data to increase reliability but it remains a scenario about a future state given an (often expansive) set of assumptions (recently beautifully visualized by Koerth, Bronner, & Mithani, 2020).
  4. Stop predicting, start communicating. Communication is pivotal during a crisis. An abundance of research shows that communicating clearly and honestly is a best practice during a crisis, generally comforting the general public (e.g., Seeger, 2006). Squazzoni et al. (2020) call for transparent communication. by stating that “[t]he limitations of models and the policy recommendations derived from them have to be openly communicated and transparently addressed”. We are united in our aim to avert the COVID-19 crisis but should be careful that overconfidence doesn’t erode society’s trust in science. Stating unequivocally that we hope – based on expectations – to avert a crisis by implementing some policy, does not preclude altering our course of action when an updated scenario about the future may require us to do so. Modellers should communicate clearly to policy-makers and the general public that this is the role of computational models that are being updated daily.

Squazzoni et al. (2020) set out the agenda for our community in the coming months and it is an important one. Let’s hope that the expectations from the scenarios in our well-informed models will not fall on deaf ears.

References

Edmonds, B., Polhill, G., & Hales, D. (2019). Predicting Social Systems – a Challenge. Review of Artificial Societies and Social Simulation, 4th June 2019. https://rofasss.org/2018/11/04/predicting-social-systems-a-challenge

Edmonds, B., Le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root, H., & Squazzoni, F. (2019). Different Modelling Purposes. Journal of Artificial Societies and Social Simulation, 22(3), 6. <http://jasss.soc.surrey.ac.uk/22/3/6.html> doi: 10.18564/jasss.3993

Elsenbroich, C., & Badham, J. (2020). Focussing on our Strengths. Review of Artificial Societies and Social Simulation, 12th April 2020.
https://rofasss.org/2020/04/12/focussing-on-our-strengths/

Google. (2020). COVID-19 Mobility Reports. https://www.google.com/covid19/mobility/ (Accessed 15th April 2020)

Hofman, J. M., Sharma, A., & Watts, D. J. (2017). Prediction and Explanation in Social Systems. Science, 355, 486–488. doi: 10.1126/science.aal3856

Koerth, M., Bronner, L., & Mithani, J. (2020, March 31). Why It’s So Freaking Hard To Make A Good COVID-19 Model. FiveThirtyEight. https://fivethirtyeight.com/

Salganik, M. J. et al. (2020). Measuring the Predictability of Life Outcomes with a Scientific Mass Collaboration. PNAS. 201915006. doi: 10.1073/pnas.1915006117

Seeger, M. W. (2006). Best Practices in Crisis Communication: An Expert Panel Process, Journal of Applied Communication Research, 34(3), 232-244.  doi: 10.1080/00909880600769944

Squazzoni, F., Polhill, J. G., Edmonds, B., Ahrweiler, P., Antosz, P., Scholz, G., Chappin, É., Borit, M., Verhagen, H., Giardini, F. and Gilbert, N. (2020) Computational Models That Matter During a Global Pandemic Outbreak: A Call to Action. Journal of Artificial Societies and Social Simulation, 23(2):10. <http://jasss.soc.surrey.ac.uk/23/2/10.html>. doi: 10.18564/jasss.4298

Van Bavel, J. J. et al. (2020). Using social and behavioural science to support COVID-19 pandemic response. PsyArXiv. https://doi.org/10.31234/osf.io/y38m9

Wynants. L., et al. (2020). Prediction models for diagnosis and prognosis of COVID-19 infection: systematic review and critical appraisal. BMJ, 369, m1328. doi: 10.1136/bmj.m1328

de Matos Fernandes, C. A. and Keijzer, M. A. (2020) No one can predict the future: More than a semantic dispute. Review of Artificial Societies and Social Simulation, 15th April 2020. https://rofasss.org/2020/04/15/no-one-can-predict-the-future/

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)