Tag Archives: policy modelling

Content

Policy modelling requires a multi-scale, multi-criteria and diverse-framing approach

2 Comments

Lessons from a session at SocSimFest 2023

By Gary Polhill and Juliette Rouchier

Bruce Edmonds organized a stimulating session at the SocSimFest held 15-16 March 2023. Entitled, “How to do wrong using Social Simulation – as a result of arrogance, laziness or ill intent.” One of the presentations (Rouchier 2023) covered the modelling used to justify lockdowns in various countries. This talk concentrated on the harms lockdowns caused and suggested that they were unnecessary; a discourse that is not the most present in the media and takes an alternative view to the idea that a scientific consensus exists in real-time and could lead to the best decision. There was some ‘vigorous’ debate afterwards, but here we expand on an important point that came out of that debate: Modelling the effects of Covid to inform policy on managing the disease requires much more than epidemiological modelling. We might speculate, then, whether in general, modelling for policy intervention means ensuring greater coverage of the wider system than might be deemed strictly necessary for the immediate policy question in hand. Though such speculation has apparent consequences for model complicatedness that go beyond Sun et al.’s (2016) ‘Medawar zone’ for empirical ABM, there is an interpretation of this requirement for extended coverage that is also compatible with preferences for simpler models.

Going beyond the immediate case of Covid would require the identification of commonalities in the processes of decision making that could be extrapolated to other situations. We are less interested in that here than making the case that simulation for policy analysis in the context of Covid entails greater coverage of the system than might be expected given the immediate questions in hand. The expertise of Rouchier means our focus is primarily on the experience of Covid in France. Generalisation of the principle to wider coverage beyond this case is a matter of conjecture that we propose making.

Handling Covid: an evaluation that is still in progress

Whether governments were right or wrong to implement lockdowns of varying severity is a matter that will be for historians to debate. During that time various researchers developed models, including agent-based models, that were used to advise policymakers on handling an emergency situation predicated on higher rates of mortality and hospitalisation.[1] Assessing the effectiveness of the lockdowns empirically would require us to be able to collect data from parallel universes in which they were not implemented. The fact that we cannot do this leaves us, as Rouchier pointed out, either comparing outcomes with models’ predictions – which is problematic if the models are not trusted – or comparing outcomes across countries with different lockdown policies – which has so far been inconclusive even if it weren’t problematic because of differences in culture and geography from one nation to another. Such comparison will nevertheless be the most fruitful in time, although the differences of implementation among countries will doubtless induce long discussions about the most important factors to consider for defining relevant Non-Pharmaceutical Interventions (NPI).[2]

The effects of the lockdowns themselves on people’s mental and physical health, child development, and on the economy and working practices, are also the subject of emerging data post-lockdown. Some of these consequences have been severe – not least for the individuals concerned. Though not germane to the central argument of this brief document, it is worth noting that the same issue with unobservable parallel universes means that scientific rather than historical assessment of whether these outcomes are better or worse than any outcomes for those individuals and society at large in the absence of lockdowns is also impossible.

For our purposes, the most significant aspect of this second point is that the discussion has arisen after the epidemic emergency: First, it is noteworthy that these matters could perfectly well have been considered in models during the crisis. Indeed, contrasting the positive effect (saving lives or saving a public service) with negative effects (children’s withdrawal from education,[3] increasing psychological distress, not to mention domestic abuse – Usta et al. 2021) is typically what cost-benefit analysis, based on multi-criteria modelling, is supposed to elicit (Roy, 1996). In modelling for public policy decision-making, it is particularly clear that there is no universally ‘superior’ or ‘optimum’ indicator to be used for comparing options; but several indicators to evaluate diverse alternative policies. A discussion about the best decision for a population has to be based on the best description of possible policies and their evaluations according to the chosen indicators (Pluchinotta et al., 2022). This means that a hierarchy of values has to be made explicit to justify the hierarchy of most important indicators. During the Covid crisis one question that could have been asked (should it not have been) is: who is the most vulnerable population to protect? Is it old people because of disease or young people because of potential threats to their future chances in life?

Second, it is clear that this answer could vary in time with information and the dynamics of variant of Covid. For example, as soon as Omicron was announced by South Africa’s doctors, it was said to be less dangerous than earlier variants.[4] In that sense, the discussion of balancing priorities, in a dynamic way, in this historical period is very typical of what could also be central in other public discussions where the whole population is facing a highly uncertain future, and where the evolution of knowledge is rapid. But it is difficult to know in advance which indicators should be considered since some signals can be very weak at some point in time, but then be confirmed as highly relevant later on – essentially this is the problem of the omitted-variable bias.

The discussion about risks to mental health was vivid in 2020 already: some psychologists were soon showing the risk for people with mental health issues or women with violent husbands;[5] while the discussion about effects on children started early in 2020 (Singh et al., 2020). However this issue only started to be considered publicly by the French government a year and a half later. One interpretation of the time differential is that the signal seemed too weak for non-specialists early on, when the specialists had already seen the disturbing signs.

In science, we have no definitive rule to decide when a weak signal at present will later turn out to be truly significant. Rather, it is ‘society’ as a whole that decides on the value of different indicators (sometimes only with the wisdom of hindsight) and scientists should provide knowledge on these. This goes back to classical questions of hierarchy of values about the diverse stakes people hold in questions that recur perennially in decision science.

Modelling for policy making: tension between complexity and elegance?

Edmonds (2022) presented a paper at SSC 2022 outlining four ‘levels’ of rigour needed when conducting social simulation exercises, reserving the highest level for using agent-based models to inform public policy. Page limitations for conference submissions meant he was unable to articulate in the paper as full a list of the stipulations for rigour in the fourth level as he was for the other three. However, Rouchier’s talk at the SocSimFest brought into sharp focus that at least one of those stipulations is that models of public policy should always have broader coverage of the system than is strictly necessary for the immediate question in hand. This has the strange-seeming consequence that exclusively epidemiological models are inadequate to the task of modelling how a contagious illness should be controlled. For any control measure that is proposed, such a stipulation entails that the model be capable of exploring not only the effect on disease spread, but also potential wider effects of relevance to societal matters generally in the domain of other government departments: such as, energy, the environment, business, justice, transportation, welfare, agriculture, immigration, and international relations.

The conjecture that for any modelling challenge in complex or wicked systems, thorough policy analysis entails broader system coverage than the immediate problem in hand (KIDS-like – see Edmonds & Moss 2005), is controversial for those who like simple, elegant, uncomplicated models (KISS-like). Worse than that, while Sun et al. (2016), for example, acknowledge that the Medawar zone for empirical models is at a higher level of complicatedness than for theoretical models, the coverage implied by this conjecture is broader still. The level of complicatedness implied will also be controversial for those who don’t mind complex, complicated models with large numbers of parameters. It suggests that we might need to model ‘everything’, or that policy models are then too complicated for us to understand, and as a consequence, perhaps using simulations to analyse policy scenarios is inappropriate. The following considers each of these objections in turn with a view to developing a more nuanced analysis of the implications of such a conjecture.

Modelling ‘everything’ is a matter that is the easiest to reject as a necessary implication of modelling ‘more things’. Modelling, say, the international relations implications of proposed national policy on managing a global pandemic, does not mean one is modelling the lifecycle of extremophile bacteria, or ocean-atmosphere interactions arising from climate change, or the influence of in-home displays on domestic energy consumption, to choose a few random examples of a myriad things that are not modelled. It is not even clear what modelling ‘everything’ really means – phenomena in social and environmental systems can be modelled at diverse levels of detail, at scales from molecular to global. Fundamentally, it is not even clear that we have anything like a perception of ‘everything’, and hence no basis for representing ‘everything’ in a model. Further, the Borges argument[6] holds in that having a model that would be the same as reality makes it useless to study as it is then wiser to study reality directly. Neither universal agreement nor objective criteria[7] exist for the ‘correct’ level of complexity and complication at which to model phenomena, but failing to engage with a broader perspective on the systemic effects of phenomena leaves one open to the kind of excoriating criticism exemplified by Keen’s (2021) attack on economists’ analysis of climate change.

At the other end of the scale, doing no modelling at all is also a mistake. As Polhill and Edmonds (2023) argue, leaving simulation models out of policy analysis essentially makes the implicit assumption that human cognition is adequate to the task of deciding on appropriate courses of action facing a complex situation. There is no reason (besides hubris) to believe that this is necessarily the case, and plenty of evidence that it is not. Not least of such evidence is that many of the difficult decisions we now face around such things as managing climate change and biodiversity have been forced upon us by poor decision-making in the past.

Cognitive constraints and multiple modellers

This necessity to consider many dimensions of social life within models that are ‘close enough’ to the reality to convince decision-makers induces a risk of ‘over’-complexity. Its main default is the building of models that are too complicated for us to understand. This is a valid concern in the sense that building an artificial system that, though simpler than the real world, is still beyond human comprehension, hardly seems a worthwhile activity. The other concern is that of the knowledge needed by the modeller: how can one person be able to imagine an integrative model which includes (for example) employment, transportation, food, schools, international economy, and any other issue which is needed for a serious analysis of the consequences of policy decisions?

Options that still entail broader coverage but not a single overcomplicated integrated model are: 1/ step-by-step increase in the complexity of the model in a community of practitioners; 2/ confrontation of different simple models with different hypotheses and questions; 3/ superposition and integration of simple models into one, through a serious work on the convergence of ontologies (with a nod to Voinov and Shugart’s (2013) warnings).

  1. To illustrate this first approach, let us stay with the case of the epidemic model. One can start with an epidemiological simulation where we fit to the fact that if we tell people to stay at home then we will cut hospitalizations by enough that health services will not be overwhelmed. But then we are worried that this might have a negative impact on the economy. So we bring in modelling components that simulate all four combinations of person/business-to-person/business transactions, and this shows that if we pay businesses to keep employees on their books, we have a chance of rebooting the economy after the pandemic is over.[8] But then we are concerned that businesses might lie about who their employees are, that office-workers who can continue to work at home are privileged over those with other kinds of job, that those with a child-caring role in their households are disadvantaged in their ability to work at home if the schools are closed, and that the mental health of those who live alone is disproportionately impacted through cutting off their only means of social intercourse. And so more modelling components are brought in. In a social context, this incremental addition of the components of a complicated model may mean it is more comprehensible to the team of modellers.

    If the policy maker really wants to increase her capacity to understand her possible actions with models, she would also have to make sure to invite several researchers for each modelled aspect, as no single social science is free of controversy, and the discussions about consequences should rely on contradictory theories. If a complex model has to be built, it can indeed propose different hypotheses on behaviours, functioning of economy, sanitary risks depending on the type of encounter.[9] It is then more of a modelling ‘framework’ with several options for running various different specific models with different implementation options. One advantage of modelling that applies even in cases where Borges argument applies, is that testing out different hypotheses is harmless for humans (unlike empirical experiments) and can produce possible futures, seen as trajectories that can then be evaluated in real time with relevant indicators. With a serious group of modellers and statisticians, providing contradicting views, not only can the model be useful for developing prospective views, but also the evaluation of hypotheses could be done rapidly.

  2. The CoVprehension Collective (2020) showed another approach, more fluid in its organisation. The idea is “one question, one model”, and the constraint is to have a pedagogic result where a simple phenomenon would be illustrated. Different modellers could realise one or several models on simple issues, so that to explain one simple phenomenon, paradox or show a tautological affirmation. In the process, the CoVprehension team would create moving sub-teams, associate on one specific issue and propose their hypotheses and results in a very simple manner. Such a protocol was purely oriented for explanation to the public, but the idea would be to organise a similar dynamic for policy makers. The system is cheap (it was self-organised with researchers and engineers, with zero funding but their salary) and it sustained lively discussions, with different points of view. Questions could go from differences between possible NPI, with an algorithmic description of these NPI that could make the understanding of processes more precise, to an explanation of the reason why French supermarkets were missing toilet paper. Twenty questions were answered in two months, thus indicating that such a working dynamic is feasible in real-time and provides useful and interesting inputs to discussion.

  3. To avoid too complicated a model, the fusion of both approaches could also be conceived: the addition of dimensions to a large central model could be first tested through simple models, the main process of explanation could be found and this process reproduced within the theoretical framework of the large model. This would constitute both a production of diversity of points of view and models and the aggregation of all points of view in one large model. The fact that the model should be large is important, as ‘size matters’ in diffusion models (e.g. Gotts & Polhill 2010), and thus simple, small models would benefit from this work as well.

As some modellers like complex models (and can think with the help of these models) and others rely on simple stories to increase their understanding of the world, only the creation of an open community of diverse specialists and modellers, KISS as well as KIDS, such a collective step-by-step elaboration could resolve the central problem that ‘too complicated to understand’ is a relative, rather than absolute, assessment. One very important prerequisite of such collaboration is that there is genuine ‘horizontality’ of the community: where each participant is listened to seriously whatever their background, which can be an issue in interdisciplinary work, especially involving people of mixed career stage. Be that as it may, the central conjecture remains: agent-based modelling for policy analysis should be expected to involve even more complicated (assemblages of) models than empirical agent-based modelling.

Endnotes

[1] This point is the one that is the most disputed ex-post in France, where lockdowns were justified (as in other countries) to “protect hospitals”. In France, the idea was not to avoid deaths of older people (90% of deaths were people older than 60, this demographic being 20% of the population), but to avoid hospitals being overwhelmed with Covid cases taking the place of others. In France, the official data regarding hospital activity states that Covid cases represented 2% of hospitalizations and 5% of Intensive Care Unit (ICU) utilizations. Further, hospitals halved their workload from March to May 2020 because of almost all surgery being blocked to keep ICUs free. (In October-December 2020, although the epidemic was more significant at that time, the same decision was not taken). Arguably, 2% of 50% not an increase that should destroy a functioning system – https://www.atih.sante.fr/sites/default/files/public/content/4144/aah_2020_analyse_covid.pdf – page 2. Fixing dysfunction in the UK’s National Health Services has been a long-standing, and somewhat tedious, political and academic debate in the country for years, even before Covid (e.g. Smith 2007; Mannion & Braithwaite 2012; Pope & Burnes 2013; Edwards & Palmer 2019).

[2] An interesting difference that French people heard about was that in the UK, people could wander on the beaches during lockdowns, whereas in France it was forbidden to go to any natural area – indeed, it was forbidden to go further than one kilometre from home. Whereas, in fact, in the UK the lockdown restrictions were a ‘devolved matter’, with slightly different policies in each of the UK’s four member nations, though very similar legislation. In England, Section 6 paragraph (1) of The Health Protection (Coronavirus, Restrictions) (England) Regulations 2020 stated that “no person may leave the place where they are living without reasonable excuse”, with paragraph (2) covering examples of “reasonable excuses” including for exercise, obtaining basic necessities, and accessing public services. Similar wording was used by other devolved nations. None of the regulations stipulated any maximum distance from a person’s residence that these activities had to take place – interpretation of the UK’s law is based on the behaviour of the ‘reasonable person’ (the so-called ‘man on the Clapham omnibus’ – see Łazowski 2021). However, differing interpretations of what ‘resonable people’ would do between the citizenry and the constabulary led to fixed penalty notices being issued for taking exercise more than five miles (eight kilometres) from home – e.g. https://www.theguardian.com/uk-news/2021/jan/09/covid-derbyshire-police-to-review-lockdown-fines-after-walkers-given-200-penalties In Scotland, though the Statutory Instrument makes no mention of any distance, people were ‘given guidance’ not to travel more than five miles from home for leisure and recreation, and were still advised to stay “within their local area” after this restriction was lifted (see https://www.gov.scot/news/travel-restrictions-lifted/).

[3] A problem which seems to be true in various countries https://www.unesco.org/en/articles/new-academic-year-begins-unesco-warns-only-one-third-students-will-return-school
https://www.kff.org/other/report/kff-cnn-mental-health-in-america-survey/
https://eu.usatoday.com/in-depth/news/health/2023/05/15/school-avoidance-becomes-crisis-after-covid/11127563002/#:~:text=School%20avoidant%20behavior%2C%20also%20called,since%20the%20COVID%2D19%20pandemic
https://www.bbc.com/news/health-65954131

[4] https://www.cityam.com/omicron-mild-compared-to-delta-south-african-doctors-say/

[5] https://www.terrafemina.com/article/coronavirus-un-psy-alerte-sur-les-risques-du-confinement-pour-la-sante-mentale_a353002/1

[6] In 1946, in El hacedor, Borges described a country where the art of building maps is so excessive in the need for details that the whole country is covered by the ideal map. This leads to obvious troubles and the disappearance of geographic science in this country.

[7] See Brewer et al. (2016) if the Akaike Information Criterion is leaping to your mind at this assertion.

[8]  Although this assumption might not be stated that way anymore, as the hypothesis that many parts of the economy would hugely suffer started to reveal its truth even before the end of the crisis: a problem that had only been anticipated by a few prominent economists (e.g. Boyer, 2020). This failure shows mainly that the description that most economists make of the economy is too simplistic – as often reproached – to be able to anticipate massive disruptions. Everywhere in the world the informal sector was almost completely stopped as people could neither work in their job nor meet for information market exchange, which causes misery for a huge part of the earth population, among the most vulnerable (ILO, 2022).

[9] A real issue that became obvious is that the nosocomial infections are (still) extremely important in hospitals, as the evaluation of the number of infections in hospitals for Covid19 are estimated to be 20 to 40% during the first epidemic (Abbas et al. 2021).

Acknowledgements

GP’s work is supported by the Scottish Government Rural and Environment Science and Analytical Services Division (project reference JHI-C5-1).

References

Abbas, M., Nunes, T. R., Martischang, R., Zingg, W., Iten, A., Pittet, D. & Harbarth, S. (2021) Nosocomial transmission and outbreaks of coronavirus disease 2019: the need to protect both patients and healthcare workers. Antimicrobial Resistance & Infection Control 10, 7. doi:10.1186/s13756-020-00875-7

Boyer, R. (2020) Les capitalismes à l’épreuve de la pandémie, La découverte, Paris.

Brewer, M., Butler, A. & Cooksley, S. L. (2016) The relative performance of AIC, AICC and BIC in the presence of unobserved heterogeneity. Methods in Ecology and Evolution 7 (6), 679-692. doi:10.1111/2041-210X.12541

the CoVprehension Collective (2020) Understanding the current COVID-19 epidemic: one question, one model. Review of Artificial Societies and Social Simulation, 30th April 2020. https://rofasss.org/2020/04/30/covprehension/

Edmonds, B. (2022) Rigour for agent-based modellers. Presentation to the Social Simulation Conference 2022, Milan, Italy. https://cfpm.org/rigour/

Edmonds, B. & Moss, S. (2005) From KISS to KIDS – an ‘anti-simplistic’ modelling approach. Lecture Notes in Artificial Intelligence 3415, pp. 130-144. doi:10.1007/978-3-540-32243-6_11

Edwards, N. & Palmer, B. (2019) A preliminary workforce plan for the NHS. British Medical Journal 365 (8203), I4144. doi:10.1136/bmj.l4144

Gotts, N. M. & Polhill, J. G. (2010) Size matters: large-scale replications of experiments with FEARLUS. Advances in Complex Systems 13 (04), 453-467. doi:10.1142/S0219525910002670

ILO Brief (2020) Impact of lockdown measures on the informal economy, https://www.ilo.org/global/topics/employment-promotion/informal-economy/publications/WCMS_743523/lang–en/index.htm

Keen, S. (2021) The appallingly bad neoclassical economics of climate change. Globalizations 18 (7), 1149-1177. doi:10.1080/14747731.2020.1807856

Łazowski, A. (2021) Legal adventures of the man on the Clapham omnibus. In Urbanik, J. & Bodnar, A. (eds.) Περιμένοντας τους Bαρβάρους. Law in a Time of Constitutional Crisis: Studies Offered to Mirosław Wyrzykowski. C. H. Beck, Munich, Germany, pp. 415-426. doi:10.5771/9783748931232-415

Mannion, R. & Braithwaite, J. (2012) Unintended consequences of performance measurement in healthcare: 20 salutary lessons from the English National Health Service. Internal Medicine Journal 42 (5), 569-574. doi:10.1111/j.1445-5994.2012.02766.x

Pluchinotta I., Daniell K.A., Tsoukiàs A. (2002), “Supporting Decision Making within the Policy Cycle: Techniques and Tools”, In M. Howlett (ed.), Handbook of Policy Tools, Routledge, London, 235 – 244. https://doi.org/10.4324/9781003163954-24.

Polhill, J. G. & Edmonds, B. (2023) Cognition and hypocognition: Discursive and simulation-supported decision-making within complex systems. Futures 148, 103121. doi:10.1016/j.futures.2023.103121

Pope, R. & Burnes, B. (2013) A model of organisational dysfunction in the NHS. Journal of Health Organization and Management 27 (6), 676-697. doi:10.1108/JHOM-10-2012-0207

Rouchier, J. (2023) Presentation to SocSimFest 23 during session ‘How to do wrong using Social Simulaion – as a result of arrogance, laziness or ill intent’. https://cfpm.org/slides/JR-Epi+newTINA.pdf

Roy, B. (1996) Multicriteria methodology for decision analysis, Kluwer Academic Publishers.

Singh, S, Roy, D., Sinha, K., Parveen, S., Sharma G. & Joshic, G. (2020) Impact of COVID-19 and lockdown on mental health of children and adolescents: A narrative review with recommendations, Psychiatry Research. 2020 Nov; 293: 113429, 10.1016/j.psychres.2020.113429

Smith, I. (2007). Breaking the dysfunctional dynamics. In: Building a World-Class NHS. Palgrave Macmillan, London, pp. 132-177. doi:10.1057/9780230589704_5

Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., Balbi, S., Nolzen, H., Müller, B., Schulze, J. & Buchmann, C. M. (2016) Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software 86, 56-67. doi:10.1016/j.envsoft.2016.09.006

Usta, J., Murr, H. & El-Jarrah, R. (2021) COVID-19 lockdown and the increased violence against women: understanding domestic violence during a pandemic. Violence and Gender 8 (3), 133-139. doi:10.1089/vio.2020.0069

Voinov, A. & Shugart, H. H. (2013) ‘Integronsters’, integral and integrated modeling. Environmental Modelling & Software 39, 149-158. doi:10.1016/j.envsoft.2012.05.014

Polhill, G. and Rouchier, J. (2023) Policy modelling requires a multi-scale, multi-criteria and diverse-framing approach. Review of Artificial Societies and Social Simulation, 31 Jul 2023. https://rofasss.org/2023/07/31/policy-modelling-necessitates-multi-scale-multi-criteria-and-a-diversity-of-framing

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

An Institute for Crisis Modelling (ICM) – Towards a resilience center for sustained crisis modeling capability

Leave a comment

By Fabian Lorig1*, Bart de Bruin2, Melania Borit3, Frank Dignum4, Bruce Edmonds5, Sinéad M. Madden6, Mario Paolucci7, Nicolas Payette8, Loïs Vanhée4

*Corresponding author
1 Internet of Things and People Research Center, Malmö University, Sweden
2 Delft University of Technology, Netherlands
3 CRAFT Lab, Arctic University of Norway, Tromsø, Norway
4 Department of Computing Science, Umeå University, Sweden
5 Centre for Policy Modelling, Manchester Metropolitan University Business School, UK
6 School of Engineering, University of Limerick, Ireland
7 Laboratory of Agent Based Social Simulation, ISTC/CNR, Italy
8 Complex Human-Environmental Systems Simulation Laboratory, University of Oxford, UK

The Need for an ICM

Most crises and disasters do occur suddenly and hit the society while it is unprepared. This makes it particularly challenging to react quick to their occurrence, to adapt to the resulting new situation, to minimize the societal impact, and to recover from the disturbance. A recent example was the Covid-19 crisis, which revealed weak points of our crisis preparedness. Governments were trying to put restrictions in place to limit the spread of the virus while ensuring the well-being of the population and at the same time preserving economic stability. It quickly became clear that interventions which worked well in some countries did not seem to have the intended effect in other countries and the reason for this is that the success of interventions to a great extent depends on individual human behavior.

Agent-based Social Simulations (ABSS) explicitly model the behavior of the individuals and their interactions in the population and allow us to better understand social phenomena. Thus, ABSS are perfectly suited for investigating how our society might be affected by different crisis scenarios and how policies might affect the societal impact and consequences of these disturbances. Particularly during the Covid-19 crisis, a great number of ABSS have been developed to inform policy making around the globe (e.g., Dignum et al. 2020, Balkely et al. 2021, Lorig et al. 2021). However, weaknesses in creating useful and explainable simulations in a short time also became apparent and there is still a lack of consistency to be better prepared for the next crisis (Squazzoni et al. 2020). Especially, ABSS development approaches are, at this moment, more geared towards simulating one particular situation and validating the simulation using data from that situation. In order to be prepared for a crisis, instead, one needs to simulate many different scenarios for which data might not yet be available. They also typically need a more interactive interface where stake holders can experiment with different settings, policies, etc.

For ABSS to become an established, reliable, and well-esteemed method for supporting crisis management, we need to organize and consolidate the available competences and resources. It is not sufficient to react once a crisis occurs but instead, we need to proactively make sure that we are prepared for future disturbances and disasters. For this purpose, we also need to systematically address more fundamental problems of ABSS as a method of inquiry and particularly consider the specific requirements for the use of ABSS to support policy making, which may differ from the use of ABSS in academic research. We therefore see the need for establishing an Institute for Crisis Modelling (ICM), a resilience center to ensure sustained crisis modeling capability.

The vision of starting an Institute for Crisis Modelling was the result of the discussions and working groups at the Lorentz Center workshop on “Agent Based Simulations for Societal Resilience in Crisis Situations” that took place in Leiden, Netherlands from 27 February to 3 March 2023**.

Vision of the ICM

“To have tools suitable to support policy actors in situations that are of
big uncertainty, large consequences, and dependent on human behavior.”

The ICM consists of a taskforce for quickly and efficiently supporting policy actors (e.g., decision makers, policy makers, policy analysts) in situations that are of big uncertainty, large consequences, and dependent on human behavior. For this purpose, the taskforce consists of a larger (informal) network of associates that contribute with their knowledge, skills, models, tools, and networks. The group of associates is composed of a core group of multidisciplinary modeling experts (ranging from social scientists and formal modelers to programmers) as well as of partners that can contribute to specific focus areas (like epidemiology, water management, etc.). The vision of ICM is to consolidate and institutionalize the use of ABSS as a method for crisis management. Although physically ABSS competences may be distributed over a variety of universities, research centers, and other institutions, the ICM serves as a virtual location that coordinates research developments and provides a basic level of funding and communication channel for ABSS for crisis management. This does not only provide policy actors with a single point of contact, making it easier for them to identify who to reach when simulation expertise is needed and to develop long-term trust relationships. It also enables us to jointly and systematically evolve ABSS to become a valuable and established tool for crisis response. The center combines all necessary resources, competences, and tools to quickly develop new models, to adapt existing models, and to efficiently react to new situations.

To achieve this goal and to evolve and establish ABSS as a valuable tool for policy makers in crisis situations, research is needed in different areas. This includes the collection, development, critical analysis, and review of fundamental principles, theories, methods, and tools used in agent-based modeling. This also includes research on data handling (analysis, sharing, access, protection, visualization), data repositories, ontologies, user-interfaces, methodologies, documentation, and ethical principles. Some of these points are concisely described in (Dignum, 2021, Ch. 14 and 15).

The ICM shall be able to provide a wide portfolio of models, methods, techniques, design patterns, and components required to quickly and effectively facilitate the work of policy actors in crisis situations by providing them with adequate simulation models. For the purpose of being able to provide specialized support, the institute will coordinate the human effort (e.g., the modelers) and have specific focus areas for which expertise and models are available. This might be, for instance, pandemics, natural disasters, or financial crises. For each of these focus areas, the center will develop different use cases, which ensures and facilitates rapid responses due to the availability of models, knowledge, and networks.

Objectives of the ICM

To achieve this vision, there are a series of objectives that a resilience center for sustained crisis modeling capability in crisis situations needs to address:

1) Coordinate and promote research

Providing quick and appropriate support for policy actors in crisis situations requires not only a profound knowledge on existing models, methods, tools, and theories but also the systematic development of new approaches and methodologies. This is to advance and evolve ABSS for being better prepared for future crises and will serve as a beacon for organizing the ABSS research oriented towards practical applications.

2) Enable trusted connections with policy actors

Sustainable collaborations and interactions with decision-makers and policy analysts as well as other relevant stakeholders is a great challenge in ABSS. Getting in contact with the right actors, “speaking the same language”, and having realistic expectations are only some of the common problems that need to be addressed. Thus, the ICM should not only connect to policy actors in times of crises, but have continuous interactions, provide sample simulations, develop use cases, and train the policy actors wherever possible.

3) Enable sustainability of the institute itself

Classic funding schemes are unfit for responding in crises, which require fast responses with always-available resources as well as the continuous build-up of knowledge, skills, network, and technological buildup requires long-term. Sustainable funding is needed that for enabling such a continuity, for which the IBM provides a demarked, unifying frame.

4) Actively maintain the network of associates

Maintaining a network of experts is challenging because it requires different competences and experiences. PhD candidates, for instance, might have a great practical experience in using different simulation frameworks, however, after their graduation, some might leave academia and others might continue to other positions where they do not have the opportunity to use their simulation expertise. Thus, new experts need to be acquired continuously to form a resilient and balanced network.

5) Inform policy actors

The most advanced and profound models cannot do any good in crisis situations in case of a lacking demand from policy actors. Many modelers perceive a certain hesitation from policy actors regarding the use of ABSS which might be due to them being unfamiliar with the potential benefits and use-cases of ABSS, lacking trust in the method itself, or simply due to a lack of awareness that ABSS actually exists. Hence, the center needs to educate policy makers and raise awareness as well as improve trust in ABSS.

6) Train the next generation of experts

To quickly develop suitable ABSS models in critical situations requires a variety of expertise. In addition to objective 4, the acquisition of associates, it is also of great importance to educate and train the next generation of experts. ABSS research is still a niche and not taught as an inherent part of the spectrum of methods of most disciplines. The center shall promote and strengthen ABSS education to ensure the training of the next generation of experts.

7) Engage the general public

Finally, the success of ABSS does not only depend on the trust of policy actors but also on how it is perceived by the general public. When developing interventions during the Covid-19 crisis and giving recommendations, the trust in the method was a crucial success factor. Also, developing realistic models requires the active participation of the general public.

Next steps

For ABSS to become a valuable and established tool for supporting policy actors in crisis situations, we are convinced that our efforts need to be institutionalized. This allows us to consolidate available competences, models, and tools as well as to coordinate research endeavors and the development of new approaches required to ensure a sustained crisis modeling capability.

To further pursue this vision, a Special Interest Group (SIG) on Building ResilienCe with Social Simulations (BRICSS) was established at the European Social Simulation Association (ESSA). Moreover, Special Tracks will be organized at the 2023 Social Simulation Conference (SSC) to bring together interested experts.

However, for this vision to become reality, the next steps towards establishing an Institute for Crisis Modelling consist of bringing together ambitious and competent associates as well as identifying core funding opportunities for the center. If the readers feel motivated to contribute in any way to this topic, they are encouraged to contact Frank Dignum, Umeå University, Sweden or any of the authors of this article.

Acknowledgements

This piece is a result of discussions at the Lorentz workshop on “Agent Based Simulations for Societal Resilience in Crisis Situations” at Leiden, NL in earlier this year! We are grateful to the organisers of the workshop and to the Lorentz Center as funders and hosts for such a productive enterprise. The final report of the workshop as well as more information can be found on the webpage of the Lorentz Center: https://www.lorentzcenter.nl/agent-based-simulations-for-societal-resilience-in-crisis-situations.html

References

Blakely, T., Thompson, J., Bablani, L., Andersen, P., Ouakrim, D. A., Carvalho, N., Abraham, P., Boujaoude, M.A., Katar, A., Akpan, E., Wilson, N. & Stevenson, M. (2021). Determining the optimal COVID-19 policy response using agent-based modelling linked to health and cost modelling: Case study for Victoria, Australia. Medrxiv, 2021-01. doi: 10.1101/2021.01.11.21249630

Dignum, F., Dignum, V., Davidsson, P., Ghorbani, A., van der Hurk, M., Jensen, M., Kammler C., Lorig, F., Ludescher, L.G., Melchior, A., Mellema, R., Pastrav, C., Vanhee, L. & Verhagen, H. (2020). Analysing the combined health, social and economic impacts of the coronavirus pandemic using agent-based social simulation. Minds and Machines, 30, 177-194. doi: 10.1007/s11023-020-09527-6

Dignum, F. (ed.). (2021) Social Simulation for a Crisis; Results and Lessons from Simulating the COVID-19 Crisis. Springer.

Lorig, Fabian, Johansson, Emil and Davidsson, Paul (2021) ‘Agent-Based Social Simulation of the Covid-19 Pandemic: A Systematic Review’ Journal of Artificial Societies and Social Simulation 24(3), 5. http://jasss.soc.surrey.ac.uk/24/3/5.html. doi: 10.18564/jasss.4601

Squazzoni, F. et al. (2020) ‘Computational Models That Matter During a Global Pandemic Outbreak: A Call to Action‘ Journal of Artificial Societies and Social Simulation 23(2), 10. http://jasss.soc.surrey.ac.uk/23/2/10.html. doi: 10.18564/jasss.4298

Lorig, F., de Bruin, B., Borit, M., Dignum, F., Edmonds, B., Madden, S.M., Paolucci, M., Payette, N. and Vanhée, L. (2023) An Institute for Crisis Modelling (ICM) –
Towards a resilience center for sustained crisis modeling capability. Review of Artificial Societies and Social Simulation, 22 May 2023. https://rofasss.org/2023/05/22/icm

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

A Tale of Three Pandemic Models: Lessons Learned for Engagement with Policy Makers Before, During, and After a Crisis

Leave a comment

By Emil Johansson1,2, Vittorio Nespeca3, Mikhail Sirenko4, Mijke van den Hurk5, Jason Thompson6, Kavin Narasimhan7, Michael Belfrage1, 2, Francesca Giardini8, and Alexander Melchior5,9

  1. Department of Computer Science and Media Technology, Malmö University, Sweden
  2. Internet of Things and People Research Center, Malmö University, Sweden
  3. Computational Science Lab, University of Amsterdam, The Netherlands
  4. Faculty of Technology, Policy and Management, Delft University of Technology, The Netherlands
  5. Department of Information and Computing Sciences, Utrecht University, The Netherlands
  6. Transport, Health and Urban Design Research Lab, The University of Melbourne, Australia
  7. Centre for Research in Social Simulation, University of Surrey, United Kingdom
  8. Department of Sociology & Agricola School for Sustainable Development, University of Groningen, The Netherlands
  9. Ministry of Economic Affairs and Climate Policy and Ministry of Agriculture, Nature and Food Quality, The Netherlands

Motivation

Pervasive and interconnected crises such as the COVID-19 pandemic, global energy shortages, geopolitical conflicts, and climate change have shown how a stronger collaboration between science, policy, and crisis management is essential to foster societal resilience. As modellers and computational social scientists we want to help. Several cases of model-based policy support have shown the potential of using modelling and simulation as tools to prepare for, learn from (Adam and Gaudou, 2017), and respond to crises (Badham et al., 2021). At the same time, engaging with policy-makers to establish effective crisis-management solutions remains a challenge for many modellers due to lacking forums that promote and help develop sustained science-policy collaborations. Equally challenging is to find ways to provide effective solutions under changing circumstances, as it is often the case with crises.

Despite the existing guidance regarding how modellers can engage with policy makers e.g. (Vennix, 1996; Voinov and Bousquet, 2010), this guidance often does not account for the urgency that characterizes crisis response. In this article, we tell the stories of three different models developed during the COVID-19 pandemic in different parts of the world. For each of the models, we draw key lessons for modellers regarding how to engage with policy makers before, during, and after crises. Our goal is to communicate the findings from our experiences to  modellers and computational scientists who, like us, want to engage with policy makers to provide model-based policy and crisis management support. We use selected examples from Kurt Vonnegut’s 2004 lecture on ‘shapes of stories’ alongside analogy with Lewis Carroll’s Alice In Wonderland as inspiration for these stories.

Boy Meets Girl (Too Late)

A Social Simulation On the Corona Crisis’ (ASSOCC) tale

The perfect love story between social modellers and stakeholders would be they meet (pre-crisis), build a trusting foundation and then, when a crisis hits, they work together as a team, maybe have some fight, but overcome the crisis together and have a happily ever after.

In the case of the ASSOCC project, we as modellers met our stakeholders too late, (i.e., while we were already in the middle of the COVID-19 crisis). The stakeholders we aimed for had already met their ‘boy’: Epidemiological modellers. For them, we were just one of the many scientists showing new models and telling them that ours should be looked at. Although, for example, our model showed that using a track and tracing-app would not help reduce the rate of new COVID-19 infections (as turned out to be the case), our psychological and social approach was novel for them. It was not the right time to explain the importance of integrating these kinds of concepts in epidemiological models, so without this basic trust, they were reluctant to work with us.

The moral of our story is that not only should we invest in a (working) relationship during non-crisis times to get the stakeholders on board during a crisis, such an approach would be helpful for us modelers too. For example, we integrated both social and epidemiological models within the ASSOCC project. We wanted to validate our model with that used by Oxford University. However, our model choices were not compatible with this type of validation. Had we been working with these types of researchers before a pandemic, we could have built a proper foundation for validation.

So, our biggest lesson learned is the importance of having a good relationship with stakeholders before a crisis hits, when there is time to get into social models and show the advantages of using these. When you invest in building and consolidating this relationship over time, we promise a happily ever after for every social modeler and stakeholder (until the next crisis hits).

Modeller’s Adventures in Wonderland

A Health Emergency Response in Interconnected Systems (HERoS) tale

If you are a modeler, you are likely to be curious and imaginative, like Alice from “Alice’s Adventures in Wonderland.” You like to think about how the world works and make models that can capture these sometimes weird mechanisms. We are the same. When Covid came, we made a model of a city to understand how its citizens would behave.

But there is more. When Alice first saw the White Rabbit, she found him fascinating. A rabbit with a pocket watch which is too late, what could be more interesting? Similarly, our attention got caught by policymakers who wear waistcoats, who are always busy but can bring change. They must need a model that we made! But why are they running away? Our model is so helpful, just let us explain! Or maybe our model is not good enough?

Yes, we fell down deep into a rabbit hole. Our first encounter with a policymaker didn’t result in a happy “yes, let’s try your model out.” However, we kept knocking on doors. How many did Alice try? But alright, there is one. It seems too tiny. We met with a group of policymakers but had only 10 minutes to explain our large-scale data-driven agent-based-like model. How can we possibly do that? Drink from a “Drink me” bottle, which will make our presentation smaller! Well, that didn’t help. We rushed over all the model complexities too fast and got applause, but that’s it. Ok, we have the next one, which will last 1 hour. Quickly! Eat an “Eat me” cake that will make the presentation longer! Oh, too many unnecessary details this time. To the next venue!

We are in the garden. The garden of crisis response. And it is full of policymakers: Caterpillar, Duchess, Cheshire Cat and Mad Hatter. They talk riddles: “We need to consult with the Head of Paperclip Optimization and Supply Management,” want different things: “Can you tell us what will be the impact of a curfew. Hmm, yesterday?” and shift responsibility from one to another. Thankfully there is no Queen of Hearts who would order to behead us.

If the world of policymaking is complex, then the world of policymaking during the crisis is a wonderland. And we all live in it. We must overgrow our obsession with building better models, learn about its fuzzy inhabitants, and find a way to instead work together. Constant interaction and a better understanding of each other’s needs must be at the centre of modeler-policymaker relations.

“But I don’t want to go among mad people,” Alice remarked.

“Oh, you can’t help that,” said the Cat: “we’re all mad here. I’m mad. You’re mad.”

“How do you know I’m mad?” said Alice.

“You must be,” said the Cat, “or you wouldn’t have come here.”

Lewis Carroll, Alice in Wonderland

Cinderella – A city’s tale

Everyone thought Melbourne was just too ugly to go to the ball…..until a little magic happened.

Once upon a time, the bustling Antipodean city of Melbourne, Victoria found itself in the midst of a dark and disturbing period. While all other territories in the great continent of Australia had ridded themselves of the dreaded COVID-19 virus, it was itself, besieged. Illness and death coursed through the land.

Shunned, the city faced scorn and derision. It was dirty. Its sisters called it a “plague state” and the people felt great shame and sadness as their family, friends and colleagues continued to fall to the virus. All they wanted was a chance to rejoin their families and countryfolk at the ball. What could they do?

Though downtrodden, the kind-hearted and resilient residents of Melbourne were determined to regain control over their lives. They longed for a glimmer of sunshine on these long, gloomy days – a touch of magic, perhaps? They turned to their embattled leaders for answers. Where was their Fairy Godmother now?

In this moment of despair, a group of scientists offered a gift in the form of a powerful agent-based model that was running on a supercomputer. This model, the scientists said, might just hold the key to transforming the fate of the city from vanquished to victor (Blakely et al., 2020). What was this strange new science? This magical black box?

Other states and scientists scoffed. “You can never achieve this!”, they said. “What evidence do you have? These models are not to be trusted. Such a feat as to eliminate COVID-19 at this scale has never been done in the history of the world!” But what of it? Why should history matter? Quietly and determinedly, the citizens of Melbourne persisted. They doggedly followed the plan.

Deep down, even the scientists knew it was risky. People’s patience and enchantment with the mystical model would not last forever. Still, this was Melbourne’s only chance. They needed to eliminate the virus so it would no longer have a grip on their lives. The people bravely stuck to the plan and each day – even when schools and businesses began to re-open – the COVID numbers dwindled from what seemed like impossible heights. Each day they edged down…

and down…

and down…until…

Finally! As the clock struck midnight, the people of Melbourne achieved the impossible: they had defeated COVID-19 by eliminating transmission. With the help of the computer model’s magic, illness and death from the virus stopped. Melbourne had triumphed, emerging stronger and more united than ever before (Thompson et al., 2022a).

From that day forth, Melbourne was internationally celebrated as a shining example of resilience, determination, and the transformative power of hope. Tens of thousands of lives were saved – and after enduring great personal and community sacrifice, its people could once again dance at the ball.

But what was the fate of the scientists and the model? Did such an experience change the way agent-based social simulation was used in public health? Not really. The scientists went back to their normal jobs and the magic of the model remained just that – magic. Its influence vanished like fairy dust on a warm Summer’s evening.

Even to this day the model and its impact largely remains a mystery (despite over 10,000 words of ODD documentation). Occasionally, policy-makers or researchers going about their ordinary business might be heard to say, “Oh yes, the model. The one that kept us inside and ruined the economy. Or perhaps it was the other way around? I really can’t recall – it was all such a blur. Anyway, back to this new social problem – Shall we attack it with some big data and ML techniques?”.

The fairy dust has vanished but the concrete remains.

And in fairness, while agent-based social simulation remains mystical and our descriptions opaque, we cannot begrudge others for ever choosing concrete over dust (Thompson et al, 2022b).

Conclusions

So what is the moral of these tales? We consolidate our experiences into these main conclusions:

  • No connection means no impact. If modellers wish for their models to be useful before, during or after a crisis, then it is up to them to start establishing a connection and building trust with policymakers.
  • The window of opportunity for policy modelling during crises can be narrow, perhaps only a matter of days. Capturing it requires both that we can supply a model within the timeframe (impossible as it may appear) and that our relationship with stakeholders is already established.
  • Engagement with stakeholders requires knowledge and skills that might be too much to ask of modelers alone, including project management, communication with individuals without a technical background, and insight into the policymaking process.
  • Being useful only sometimes means being excellent. A good model is one that is useful. By investing more in building relationships with policymakers and learning about each other, we have a bigger chance of providing the needed insight. Such a shift, however, is radical and requires us to give up our obsession with the models and engage with the fuzziness of the world around us.
  • If we cannot communicate our models effectively, we cannot expect to build trust with end-users over the long term, whether they be policy-makers or researchers. Individual models – and agent-based social simulation in general – needs better understanding that can only be achieved through greater transparency and communication, however that is achieved.

As taxing, time-consuming and complex as the process of making policy impact with simulation models might be, it is very much a fight worth fighting; perhaps even more so during crises. Assuming our models would have a positive impact on the world, not striving to make this impact could be considered admitting defeat. Making models useful to policymakers starts with admitting the complexity of their environment and willingness to dedicate time and effort to learn about it and work together. That is how we can pave the way for many more stories with happy endings.

Acknowledgements

This piece is a result of discussions at the Lorentz workshop on “Agent Based Simulations for Societal Resilience in Crisis Situations” at Leiden, NL in earlier this year! We are grateful to the organisers of the workshop and to the Lorentz Center as funders and hosts for such a productive enterprise.

References

Adam, C. and Gaudou, B. (2017) ‘Modelling Human Behaviours in Disasters from Interviews: Application to Melbourne Bushfires’ Journal of Artificial Societies and Social Simulation 20(3), 12. http://jasss.soc.surrey.ac.uk/20/3/12.html. doi: 10.18564/jasss.3395

Badham, J., Barbrook-Johnson, P., Caiado, C. and Castellani, B. (2021) ‘Justified Stories with Agent-Based Modelling for Local COVID-19 Planning’ Journal of Artificial Societies and Social Simulation 24 (1) 8 http://jasss.soc.surrey.ac.uk/24/1/8.html. doi: 10.18564/jasss.4532

Crammond, B. R., & Kishore, V. (2021). The probability of the 6‐week lockdown in Victoria (commencing 9 July 2020) achieving elimination of community transmission of SARS‐CoV‐2. The Medical Journal of Australia, 215(2), 95-95. doi:10.5694/mja2.51146

Thompson, J., McClure, R., Blakely, T., Wilson, N., Baker, M. G., Wijnands, J. S., … & Stevenson, M. (2022). Modelling SARS‐CoV‐2 disease progression in Australia and New Zealand: an account of an agent‐based approach to support public health decision‐making. Australian and New Zealand Journal of Public Health, 46(3), 292-303. doi:10.1111/1753-6405.13221

Thompson, J., McClure, R., Scott, N., Hellard, M., Abeysuriya, R., Vidanaarachchi, R., … & Sundararajan, V. (2022). A framework for considering the utility of models when facing tough decisions in public health: a guideline for policy-makers. Health Research Policy and Systems, 20(1), 1-7. doi:10.1186/s12961-022-00902-6

Voinov, A., & Bousquet, F. (2010). Modelling with stakeholders. Environmental modelling & software, 25(11), 1268-1281. doi:10.1016/j.envsoft.2010.03.007

Vennix, J.A.M. (1996). Group Model Building: Facilitating Team Learning Using System Dynamics. Wiley.

Vonnegut, K. (2004). Lecture to Case College. https://www.youtube.com/watch?v=4_RUgnC1lm8

Johansson,E., Nespeca, V., Sirenko, M., van den Hurk, M., Thompson, J., Narasimhan, K., Belfrage, M., Giardini, F. and Melchior, A. (2023) A Tale of Three Pandemic Models: Lessons Learned for Engagement with Policy Makers Before, During, and After a Crisis. Review of Artificial Societies and Social Simulation, 15 Mar 2023. https://rofasss.org/2023/05/15/threepandemic

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)