How Can ABM Models Become Part of the Policy-Making Process in Times of Emergencies – The S.I.S.A.R. Epidemic Model

By Gianpiero Pescarmona1, Pietro Terna2,*, Alberto Acquadro1, Paolo Pescarmona3, Giuseppe Russo4, and Stefano Terna5

*Corresponding author, 1University of Torino, IT, 2University of Torino, IT, retired & Collegio Carlo Alberto, IT, 3University of Groningen, NL, 4Centro Einaudi, Torino, IT, 5tomorrowdata.io

(A contribution to the: JASSS-Covid19-Thread)

We propose an agent-based model to simulate the Covid-19 epidemic diffusion, with Susceptible, Infected, symptomatic, asymptomatic, and Recovered people: hence the name S.I.s.a.R. The scheme comes from S.I.R. models, with (i) infected agents categorized as symptomatic and asymptomatic and (ii) the places of contagion specified in a detailed way, thanks to agent-based modeling capabilities. The infection transmission is related to three factors: the infected person’s characteristics and the susceptible one, plus those of the space in which contact occurs. The asset of the model is the development of a tool that allows analyzing the contagions’ sequences in simulated epidemics and identifying the places where they occur.

The characteristics of the S.I.s.a.R. model

S.I.s.a.R. can be found at https://terna.to.it/simul/SIsaR.html with information on model construction, the draft of a paper also reporting results, and an online executable version of the simulation program, built using NetLogo. The model includes the structural data of Piedmont, an Italian region, but it can be readily calibrated for other areas. The model reproduces a realistic calendar (e.g., national or local government decisions), via a dedicated script interpreter.

Why another model? The starting point has been the need to model the pandemic problem in a multi-scale way. This was initiated a few months before the publication of new frontier articles, such as Bellomo et al. (2020), so when equation-based S.I.R. models, with their different versions, were predominating.

As any model, also this one is based on some assumptions: time will tell whether these were reasonable hypotheses. Modeling the Covid-19 pandemic requires a scenario and the actors. As in a theatre play, the author defines the roles of the actors and the environment. The characters are not real, they are pre-built by the author, and they act according to their peculiar constraints. If the play is successful, it will run for a long time, even centuries. If not, we will rapidly forget it. Shakespeare’s Hamlet is still playing after centuries, even if the characters and the plot are entirely imaginary. The same holds for our simulations: we are the authors, we arbitrarily define the characters, we force them to act again and again in different scenarios. However, in our model, the micro-micro assumptions are not arbitrary but based on scientific hypotheses at the molecular level, the micro agents’ behaviors are modeled in an explicit and realistic way. In both plays and simulations, we compress the time: a whole life to 2 or 3 hours on the stage. In a few seconds, we run the Covid-19 pandemic spread in a given regional area.

With our model, we move from a macro compartmental vision to a meso and microanalysis capability. Its main characteristics are:

  • scalability: we take in account the interactions between virus and molecules inside the host, the interactions between individuals in more or less restricted contexts, the movement between different environments (home, school, workplace, open spaces, shops, in a second version, we will add transportations and long trips between regions/countries; discotheques; other social aggregation events, as football matches); the movements occur in different parts of the daily life, as in Ghorbani et al. (2020);

the scales are:

    • micro, with the internal biochemical mechanism involved in reacting to the virus, as in Silvagno et al. (2020), from where we derive the critical importance assigned to an individual intrinsic susceptibility related to the age and previous morbidity episodes; the model incorporates the medical insights of one of its co-authors, former full professor of clinical biochemistry, signing also the quoted article; a comment on Lancet (Horton, 2020) consistently signals the syndemic character of the current event: «Two categories of disease are interacting within specific populations—infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and an array of non-communicable diseases (NCDs)»;
    • meso, with the open and closed contexts where the agents behave, as reported above;
    • macro, with the emergent effects of the actions of the agents; this final analysis is a premise to evaluate the costs and benefits of the different intervention policies;
  • granularity: at any level, the interactions are partially random and therefore the final results always reflect the sum of the randomness at the different levels; changing the constraints at different levels and running multiple simulations should allow the identification of the most critical points, i.e., those on which the intervention should be focused.

Contagion sequences as a source of suggestions for intervention policies

All the previous considerations are not exhaustive. The critical point that makes helpful the production of a new model is creating a tool that allows analyzing the contagions’ sequences in simulated epidemics and identifying the places where they occur. We represent each infecting agent as a horizontal segment with a vertical connection to another agent receiving the infection. We represent the second agent via a further segment at an upper layer. With colors, line thickness, and styles, we display multiple data.

As an example, look at Fig.4: we start with two agents coming from the outside, with black as color code (external place), the first one–regular, as reported by the thickness of the segment, starting at day 0 and finishing at day 22–is asymptomatic (dashed line) and infects five agents; the second one–robust, as reported by the thickness of the segment, starting at day 0 and finishing at day 15–is asymptomatic (dashed line) and infects no one; the first of the five infected agents received the infection at home (cyan color) and turns to be asymptomatic after a few days of incubation (dotted line), and so on. Solid lines identify symptomatic agents; brown color refers to workplaces, orange to nursing homes; yellow to schools; pink to hospitals; gray to open spaces. Thick or extra-thick lines refer to fragile or extra-fragile agents, respectively.

This technique enables understanding at a glance how an epidemic episode is developing. In this way, it is easier to reason about countermeasures and, thus, to develop intervention policies. In Figs. 1-4, we can look both at the places where contagions occur and at the dynamics emerging with different levels of intervention. In Fig. 1 we find evidence of the role of the workplaces in diffusing the infection, with a relevant number of infected fragile workers. In Fig. 2, by isolating fragile workers at home, the epidemics seems to finish, but in Fig. 3, we see a thin event (a single case of contagion) that creates a bridge toward a second wave. Finally, in Fig. 4, we see that the epidemic is under control by isolating the workers and any kind of fragile agents. (Please enlarge the on-screen images to see more details).

A scatter graph showing an epidemic with regular containment measures, showing a highly significant effect of workplaces (brown)Figure 1 – An epidemic with regular containment measures, showing a highly significant effect of workplaces (brown)

A scatter graph showing The effects of stopping fragile workers at day 20, with a positive result, but home contagions (cyan) keep alive the pandemic, exploding again in workplaces (brown)

Figure 2 – The effects of stopping fragile workers at day 20, with a positive result, but home contagions (cyan) keep alive the pandemic, exploding again in workplaces (brown)

A scatter graph showing The effects of stopping fragile workers at day 20, with a positive result, but home contagions (cyan) keep alive the pandemic, exploding again in workplaces (brown)

Figure 3 – Same, analyzing the first 200 infections with evidence of the event around day 110 with the new phase due to a unique asymptomatic worker

A scatter graph showing the impoaact of Stopping fragile workers plus any case of fragility at day 15, also isolating nursing homes

Figure 4 – Stopping fragile workers plus any case of fragility at day 15, also isolating nursing homes

Batches of simulation runs

The sequence in the steps described by the four figures is only a snapshot, a suggestion. We need to explore systematically the introduction of factual, counterfactual, and prospective interventions to control the spread of the contagions. Each simulation run–whose length coincides with the disappearance of symptomatic or asymptomatic contagion cases–is a datum in a wide scenario of variability in time and effects. Consequently, we need to represent compactly the results emerging from batches of repetitions, to compare the consequences of each batch’s basic assumptions.

For this purpose, we used blocks of one thousand repetitions. Besides summarizing the results with the usual statistical indicators, we adopted the technique of the heat-maps. In this perspective, with Steinmann et al. (2020), we developed a tool for comparative analyses, not for forecasting. This consideration is consistent with the enormous standard deviation values that are intrinsic to the problem.

Figs. 5-6 provide two heat-maps reporting the duration of each simulated epidemic in the x axis and the number of the symptomatic, asymptomatic, and deceased agents in the y axis. 1,000 runs in both cases.

The actual data for Piedmont, where the curve of the contagions flattened with the end of May, with around 30 thousand subjects, is included in the cell in the first row, immediately to the right of the mode in Fig. 6. In the Fall, a second wave seems possible, jumping into one of the events of the range of events on the right side of the same figure.

Figure 5 – 1000 Epidemics without containment measures (2D histogram of (Symptomatic+Asymptomatic+Deceased against days)

Figure 6 – 1000 Epidemics with basic non-pharmaceutical containment measures, no school in September 2020 (2D histogram of (Symptomatic+Asymptomatic+Deceased against days)

In Table 1 we have a set of statistical indicators related to 1,000 runs of the simulation with the different initial conditions. Cases 1 and 2 are those of Fig. 5 and 6. Then we introduce Case 4, excluding from the workplace workers with health fragilities, so highly susceptible to contagion, with smart work when possible or sick pay conditions. The gain in the reduction of affected people and duration is relevant and increases – in Case 5 – if we leave at home all kinds of fragile people.

Scenarios Total symptomatic Total symptomatic, asymptomatic, deceased Days
1. no control 851.12
(288.52)
2253.48
(767.58)
340.10
(110.21)
2. basic controls, no school in Sep 2020 158.55
(174.10)
416.98
(462.94)
196.97
(131.18)
4. basic controls, stop fragile workers, no schools in Sep 2020 120.17
(149.10)
334.68
(413.90)
181.10
(125.46)
5. basic controls, stop fragile workers & fragile people, nursing-homes isolation, no schools in Sep 2020 105.63
(134.80)
302.62
(382.14)
174.39
(12.82)
7. basic controls, stop f. workers & fragile people, nursing-homes isolation, open factories, schools in Sep 2020 116.55
(130.91)
374.68
(394.66)
195.28
(119.33)

Table 1 – Statistical indicators, limited to the mean and to the standard deviation, reported in parentheses, for a set of experiments; the row numbers are consistent with the paper at https://terna.to.it/simul/SIsaR.html where we report a larger number of simulation experiments

In Case 7, we show that keeping the conditions of Case 5, while opening schools and factories (work places in general), increases in a limited way the adverse events.

A second version

A second version of the model is under development, using https://terna.github.io/SLAPP/, a Python shell for ABM prepared by one of the authors of this note, referring to the pioneering proposal http://www.swarm.org of the Santa Fe Institute.

References

Bellomo, N., Bingham, R., Chaplain, M. A. J., Dosi, G., Forni, G., Knopoff, D. A., Lowengrub,, J., Twarock, R., and Virgillito, M. E. (2020). A multi-scale model of virus pandemic: Heterogeneous interactive entities in a globally connected world. arXiv e-prints, art. arXiv:2006.03915, June.

Ghorbani, A., Lorig, F., de Bruin, B., Davidsson, P., Dignum, F., Dignum, V., van der Hurk, M., Jensen, M., Kammler, C., Kreulen, K., et al. (2020). The ASSOCC Simulation Model: A Response to the Community Call for the COVID-19 Pandemic. Review of Artificial Societies and Social Simulation. URL https://rofasss.org/2020/04/25/the-assocc-simulation-model/.

Horton, R. (2020). Offline: Covid-19 is not a pandemic. Lancet (London, England), 396(10255):874. URL https://www.thelancet.com/action/showPdf?pii=S0140-6736%2820%2932000-6.

Silvagno, F., Vernone, A. and Pescarmona, G. P. (2020). The Role of Glutathione in Protecting against the Severe Inflammatory Response Triggered by COVID-19. In «Antioxidants», vol. 9(7), p. 624. http://dx.doi.org/10.3390/antiox9070624.

Steinmann P., Wang J. R., van Voorn G. A., and Kwakkel J. H. (2020). Don’t try to predict covid-19. if you must, use deep uncertainty methods. Review of Artificial Societies and Social Simulation, 17. https://rofasss.org/2020/04/17/deep-uncertainty/.


Pescarmona, G., Terna, P., Acquadro, A., Pescarmona, P., Russo, G., and Terna, S. (2020) How Can ABM Models Become Part of the Policy-Making Process in Times of Emergencies - The S.I.S.A.R. Epidemic Model. Review of Artificial Societies and Social Simulation, 20th Oct 2020. https://rofasss.org/2020/10/20/sisar/


 

Flu and Coronavirus Simulator – A geospatial agent-based simulator for analyzing COVID-19 spread and public health measures on local regions

By Imran Mahmood

A Summary of: Mahmood et al. (2020)

In this paper we reviewed the lessons learned during the development of the ‘Flu and Coronavirus Simulator’ (FACS) and compare our chosen Agent-based Simulation approach with the conventional disease modelling approaches.

FACS provides an open-ended platform for the specification and implementation of the primary components of Agent-Based Simulation (ABS): (i) Agents; (ii) Virtual environment and (iii) Rule-set using a systematic Simulation Development Approach. FACS inherits features of a comprehensive simulation framework from its ancestors: (i) FLEE (Groen & Arabnejad, 2015) and (ii) FabSim3 (Groen & Arabnejad 2014). Where, FLEE mainly specializes in ABS complex dynamics e.g., agent movements; FabSim3 provides the ability to simulate a large population of agents with microscopic details using remote supercomputers. The combination of this legacy code offers numerous benefits including high performance, high scalability, and greater re-usability through a model coupling. Hence it provides an open-ended API for modellers and programmers to use it for further scientific research and development. FACS generalizes the process of disease modelling and provides a template to model any infectious disease. Thus allowing: (i) non-programmers (e.g., epidemiologists and healthcare data scientists) to use the framework as a disease modelling suite; and (ii) providing an open-ended API for modellers and programmers to use it for further scientific research and development. FACS offers a built-in location graph construction tool that allows the import of large spatial data-sets (e.g., Open Street Map), automated parsing and pre-processing of the spatial data, and generating buildings of various types, thus allowing ease in the synthesis of the virtual environment for the region under consideration. FACS provides a realistic disease transmission algorithm with the ability to capture population interactions and demographic patterns e.g., age diversity, daily life activities, mobility patterns, exposure at the street-level or in public transportation, use, or no use of face mask, assumptions of exposure within closed quarters.

We believe our approach has proven to be quite productive in modelling complex systems like epidemic spread in large regions due to ever-changing model requirements, multi-resolution abstraction, non-linear system dynamics, rule-based heuristics, and above all large-scale computing requirements. During the development of this framework, we learned that the real-world abstraction changes more rapidly than in other circumstances. For instance, the concept of social distancing and lockdown scenarios have evolved significantly since early March. Therefore, rapid changes in the ABS model were necessary. Model building in these cases benefits more from using a bottom-up approach like ABS, as opposed to any centralized analytical solution.

References

Groen, D., & Arabnejad, H. (2014). Fabsim3. GitHub. https://github.com/djgroen/FabSim3

Groen, D., & Arabnejad, H. (2015). Flee. GitHub. https:// github.com/djgroen/flee

Mahmood, I.,  Arabnejad, H., Suleimenova, D., Sassoon, I.,  Marshan, A.,  Serrano-Rico, A.,  Louvieris, P., Anagnostou, A., Taylor, S.J.E., Bell, D. & Groen, D. (2020) FACS: a geospatial agent-based simulator for analysing COVID-19 spread and public health measures on local regions, Journal of Simulation, DOI: 10.1080/17477778.2020.1800422


Mahmood, I. (2020) Flu and Coronavirus Simulator - A geospatial agent-based simulator for analyzing COVID-19 spread and public health measures on local regions. Review of Artificial Societies and Social Simulation, 10th Sept 2020. https://rofasss.org/2020/09/10/facs/


 

The role of population scale in compartmental models of COVID-19 transmission

By Christopher J. Watts1,*, Nigel Gilbert2, Duncan Robertson3, 4, Laurence T. Droy5, Daniel Ladley6and Edmund Chattoe-Brown5

*Corresponding author, 12 Manor Farm Cottages, Waresley, Sandy, SG19 3BZ, UK, 2Centre for Research in Social Simulation (CRESS), University of Surrey, Guildford GU2 7XH, UK, 3School of Business and Economics, Loughborough University, Loughborough, UK, 4St Catherine’s College, University of Oxford, Oxford, UK, 5School of Media, Communication and Sociology, University of Leicester, UK, 6University of Leicester School of Business, University of Leicester, Leicester, LE17RH, UK

(A contribution to the: JASSS-Covid19-Thread)

Compartmental models of COVID-19 transmission have been used to inform policy, including the decision to temporarily reduce social contacts among the general population (“lockdown”). One such model is a Susceptible-Exposed-Infectious-Removed (SEIR) model developed by a team at the London School of Hygiene and Tropical Medicine (hereafter, “the LSHTM model”, Davies et al., 2020a). This was used to evaluate the impact of several proposed interventions on the numbers of cases, deaths, and intensive care unit (ICU) hospital beds required in the UK. We wish here to draw attention to behaviour common to this and other compartmental models of diffusion, namely their sensitivity to the size of the population simulated and the number of seed infections within that population. This sensitivity may compromise any policy advice given.

We therefore describe below the essential details of the LSHTM model, our experiments on its sensitivity, and why they matter to its use in policy making.

The LSHTM model

Compartmental models of disease transmission divide members of a population according to their disease states, including at a minimum people who are “susceptible” to a disease, and those who are “infectious”. Susceptible individuals make social contact with others within the same population at given rates, with no preference for the other’s disease state, spatial location, or social networks (the “universal mixing” assumption). Social contacts result in infections with a chance proportional to the fraction of the population who are currently infectious. Perhaps to reduce the implausibility of the universal mixing assumption, the LSHTM model is run for each of 186 county-level administrative units (“counties”, having an average size of 357,000 people), instead of a single run covering the whole UK population (66.4 million). Each county receives the same seed infection schedule: two new infections per day for 28 days. The 186 county time series are then summed to form a time series for the UK. There are no social contacts between counties, and the 186 county-level runs are independent of each other. Outputs from the model include total and peak cases and deaths, ICU and non-ICU hospital bed occupancy, and the time to peak cases, all reported for the UK as a whole.

Interventions are modelled as 12-week reductions in contact rates, and, in the first experiment, scheduled to commence 6 weeks prior to the peak in UK cases with no intervention. Further experiments shift the start of the intervention, and trigger the intervention upon reaching a given number of ICU beds, rather than a specific time.

Studying sensitivity to population size

The 186 counties vary in their population sizes, from Isles of Scilly (2,242 people) to West Midlands (2.9 million). We investigated whether the variation in population size led to differences in model behaviour. The LSHTM model files were cloned from https://github.com/cmmid/covid-UK , while the data analysis was performed using our own scripts posted at https://github.com/innovative-simulator/PopScaleCompartmentModels .

A graph showing Peak week infections against population size (on a log scale). The peak week looks increasing linear (with the log population scale), but there is a uniform increase in peak week with more seed infections.The figure above shows the results of running the LSHTM model with populations of various sizes, each point being an average of 10 repetitions. The time, in weeks, to the peak in cases forms a linear trend with the base-10 logarithm of population. A linear regression line fitted to these points gives Peak Week = 2.70 log10(Population) – 2.80, with R2 = 0.999.

To help understand this relationship, we then compared the seeding used by the LSHTM team, i.e. 2 infectious persons per day for 28 days, to two forms of reduced seeding, 1 per day for 28 days, and 2 per day for 14 days. Halving the seeding is similar in effect, but not identical to, doubling the population size.

Deterministic versions of other compartmental models of transmission (SIR, SEIR, SI) confirmed the relation between population size and time of occurrence to be a common feature of such models. See the R and Excel files at: https://github.com/innovative-simulator/PopScaleCompartmentModels .

For the simplest, the SI model, the stock of infectious people is described by the logistic function.I(t)=N/(1+exp(-u*C*(t-t*)))Here N is the population size, u susceptibility, and C the contact rate. If I(0)=s, the number of seed infections, then it can be shown that the peak in new infections, I(t*), occurs at timet*=ln(N/s-1)/(u*C)

Hence, for N/s >> 1, the time to peak cases, t*, correlates well with log10N/s.

As well as peak cases, analogous sensitivity was found for the timing of peaks in infections and hospital admissions, and for reaching critical levels, such as the hospital bed capacity as a proportion of the population. In contrast, the heights of peaks, and totals of cases, deaths and beds were constant percentages of population when population size was varied.

Why the unit of population matters

Davies et al. (2020a) make forecasts of both the level of peak cases and the timing of their occurrence. Despite showing that two counties can vary in their results (Davies et al., 2020a, p. 6), and mentioning in the supplementary material some effects of changing the seeding schedule (Davies et al., 2020b, p. 5), they do not mention any sensitivity to population size. But, as we have shown here, given the same number and timing of seed infections, the county with the smallest population will peak in cases earlier than the one with the largest. This sensitivity to population size affects the arguments of Davies et al. in several ways.

Firstly, Davies et al. produce their forecasts for the UK by summing county-level time series. But counties with out-of-sync peaks will sum to produce a shorter, flatter peak for the UK, than would have been achieved by synchronous county peaks. Thus the forecasts of peak cases for the UK are being systematically biased down.

Secondly, timing is important for the effectiveness of the interventions. As Davies et al. note in relation to their experiment on shifting the start time of the intervention, an intervention can be too early or too late. It is too early if, when it ends after 12 weeks, the majority of the population is still susceptible to any remaining infectious cases, and a serious epidemic can still occur. At the other extreme, an intervention can be too late if it starts when most of the epidemic has already occurred.

A timing problem also threatens if the intervention is triggered by the occupancy of ICU beds reaching some critical level. This level will be reached for the UK or average county later than for a small county. Thus the problem extends beyond the timing of peaks to affect other aspects of a policy supported by the model.

Our results imply that an intervention timed optimally for a UK-level, or average county-level, cases peak, as well as an intervention triggered by a UK-level beds occupancy threshold, may be less effective for counties with far-from-average sizes.

There are multiple ways of resolving these issues, including re-scaling seed infections in line with size of population unit, simulating the UK directly rather than as a sum of counties, and rejecting compartmental models in favour of network- or agent-based models. A discussion of the respective pros and cons of these alternatives requires a longer paper. For now, we note that compartmental models remain quick and cheap to design, fit, and study. The issues with Davies et al. (2020a) we have drawn attention to here highlight (1) the importance of adequate sensitivity testing, (2) the need for care when choosing at which scale to model and how to seed an infection, and (3) the problems that can stem from uniform national policy interventions, rather than ones targeted at a more local level.

References

Davies, N. G., Kucharski, A. J., Eggo, R. M., Gimma, A., Edmunds, W. J., Jombart, T., . . . Liu, Y. (2020a). Effects of non-pharmaceutical interventions on COVID-19 cases, deaths, and demand for hospital services in the UK: a modelling study. The Lancet Public Health, 5(7), e375-e385. doi:10.1016/S2468-2667(20)30133-X

Davies, N. G., Kucharski, A. J., Eggo, R. M., Gimma, A., Edmunds, W. J., Jombart, T., . . . Liu, Y. (2020b). Supplement to Davies et al. (2020b). https://www.thelancet.com/cms/10.1016/S2468-2667(20)30133-X/attachment/cee85e76-cffb-42e5-97b6-06a7e1e2379a/mmc1.pdf


Watts, C.J., Gilbert, N., Robertson, D., Droy, L.T., Ladley, D and Chattoe-Brown, E. (2020) The role of population scale in compartmental models of COVID-19 transmission. Review of Artificial Societies and Social Simulation, 14th August 2020. https://rofasss.org/2020/08/14/role-population-scale/


 

A Bibliography of ABM Research Explicitly Comparing Real and Simulated Data for Validation

By Edmund Chattoe-Brown

The Motivation

Research that confronts models with data is still sufficiently rare that it is hard to get a representative sense of how it is done and how convincing the results are simply by “background reading”. One way to advance good quality empirical modelling is therefore simply to make it more visible in quantity. With this in mind I have constructed (building on the work of Angus and Hassani-Mahmooei 2015) the first version of a bibliography listing all ABM attempting empirical validation in JASSS between 1998 and 2019 (along with a few other example) – which generates 68 items in all. Each entry gives a full reference and also describes what comparisons are made and where in the article they occur. In addition the document contains a provisional bibliography of articles giving advice or technical support to validation and lists three survey articles that categorise large samples of simulations by their relationships to data (which served as actual or potential sources for the bibliography).

With thanks to Bruce Edmonds, this first version of the bibliography has been made available as a Centre for Policy Modelling Discussion Paper CPM-20-216, which can be downloaded http://cfpm.org/discussionpapers/256.

The Argument

It may seem quite surprising to focus only on validation initially but there is an argument (Chattoe-Brown 2019) which says that this is a more fundamental challenge to the quality of a model than calibration. A model that cannot track real data well, even when its parameters are tuned to do so is clearly a fundamentally inadequate model. Only once some measure of validation has been achieved can we decide how “convincing” it is (comparing independent empirical calibration with parameter tuning for example). Arguably, without validation, we cannot really be sure whether a model tells us anything about the real world at all (no matter how plausible any narrative about its assumptions may appear). This can be seen as a consequence of the arguments about complexity routinely made by ABM practitioners as the plausibility of the assumptions does not map intuitively onto the plausibility of the outputs.

The Uses

Although these are covered in the preface to the bibliography in greater detail, such a sample has a number of scientific uses which I hope will form the basis for further research.

  • To identify (and justify) good and bad practice, thus promoting good practice.
  • To identify (and then perhaps fill) gaps in the set of technical tools needed to support validation (for example involving particular sorts of data).
  • To test the feasibility and value of general advice offered on validation to date and refine it in the face of practical challenges faced by analysis of real cases.
  • To allow new models to demonstrably outperform the levels of validation achieved by existing models (thus creating the possibility for progressive empirical research in ABM).
  • To support agreement about the effective use of the term validation and to distinguish it from related concepts (like verification) and potentially unhelpful (for example ambiguous or rhetorically loaded) uses

The Plan

Because of the labour involved and the diversity of fields in which ABM have now been used over several decades, an effective bibliography on this kind cannot be the work of a single author (or even a team of authors). My plan is thus to solicit (fully credited) contributions and regularly release new versions of the bibliography – with new co-authors as appropriate. (This publishing model is intended to maintain the quality and suitability for citation of the resulting document relative to the anarchy that sometimes arises in genuine communal authorship!) All of the following contributions will be gratefully accepted for the next revision (on which I am already working myself in any event)

  • References to new surveys or literature reviews that categorise significant samples of ABM research by their relationship to data.
  • References for proposed new entries to the bibliography in as much detail as possible.
  • Proposals to delete incorrectly categorised entries. (There are a small number of cases where I have found it very difficult to establish exactly what the authors did in the name of validation, partly as a result of confusing or ambiguous terminology.)
  • Proposed revisions to incorrect or “unfair” descriptions of existing entries (ideally by the authors of those pieces).
  • Offers of collaboration for a proposed companion bibliography on calibration. Ultimately this will lead to a (likely very small) sample of calibrated and validated ABM (which are often surprisingly little cited given their importance to the credibility of the ABM “project” – see, for example, Chattoe-Brown (2018a, 2018b).

References

Angus, Simon D. and Hassani-Mahmooei, Behrooz (2015) ‘“Anarchy” Reigns: A Quantitative Analysis of Agent-Based Modelling Publication Practices in JASSS, 2001-2012’, Journal of Artificial Societies and Social Simulation, 18(4), October, article 16. <http://jasss.soc.surrey.ac.uk/18/4/16.html> doi:10.18564/jasss.2952

Chattoe-Brown, Edmund (2018a) ‘Query: What is the Earliest Example of a Social Science Simulation (that is Nonetheless Arguably an ABM) and Shows Real and Simulated Data in the Same Figure or Table?’ Review of Artificial Societies and Social Simulation, 11 June. https://rofasss.org/2018/06/11/ecb/

Chattoe-Brown, Edmund (2018b) ‘A Forgotten Contribution: Jean-Paul Grémy’s Empirically Informed Simulation of Emerging Attitude/Career Choice Congruence (1974)’, Review of Artificial Societies and Social Simulation, 1 June. https://rofasss.org/2018/06/01/ecb/

Chattoe-Brown, Edmund (2019) ‘Agent Based Models’, in Atkinson, Paul, Delamont, Sara, Cernat, Alexandru, Sakshaug, Joseph W. and Williams, Richard A. (eds.) SAGE Research Methods Foundations. doi:10.4135/9781526421036836969


Chattoe-Brown, E. (2020) A Bibliography of ABM Research Explicitly Comparing Real and Simulated Data for Validation. Review of Artificial Societies and Social Simulation, 12th June 2020. https://rofasss.org/2020/06/12/abm-validation-bib/


 

Basic Modelling Hygiene – keep descriptions about models and what they model clearly distinct

By Bruce Edmonds

The essence of a model is that it relates to something else – what it models – even if this is only a vague or implicit mapping. Otherwise a model would be indistinguishable from any other computer code, set of equations etc (Hesse 1964; Wartofsky 1966). The centrality of this essence makes it unsurprising that many modellers seem to conflate the two.

This is made worse by three factors.

  1. A strong version of Kuhn’s “Spectacles” (Kuhn 1962) where the researcher goes beyond using the model as a way of thinking about the world to projecting their model onto the world, so they see the world only through that “lens”. This effect seems to be much stronger for simulation modelling due to the intimate interaction that occurs over a period of time between modellers and their model.
  2. It is a natural modelling heuristic to make the model more like what it models (Edmonds & al. 2019), introducing more elements of realism. This is especially strong with agent-based modelling which lends itself to complication and descriptive realism.
  3. It is advantageous to stress the potential connections between a model (however abstract) and possible application areas. It is common to start an academic paper with a description of a real-world issue to motivate the work being reported on; then (even if the work is entirely abstract and unvalidated) to suggest conclusions for what is observed. A lack of substantiated connections between model and any empirical data can be covered up by slick passing from the world to the model and back again and a lack of clarity as to what their research achieves (Edmonds & al. 2019).

Whatever the reasons the result is similar – that the language used to describe entities, processes and outcomes in the model is the same as that used for its descriptions of what is intended to be modelled.

Such conflation is common in academic papers (albeit to different degrees). Expert modellers will not usually be confused by such language because they understand the modelling process and know what to look for in a paper. Thus one might ask, what is the harm of a little rhetoric and hype in the reporting of models? After all, we want modellers to be motivated and should thus be tolerant of their enthusiasm. To show the danger I will thus look at an example that talks about modelling aspects of ethnocentrism.

In their paper, entitled “The Evolutionary Dominance of Ethnocentric Cooperation“, Hartshorn, Kaznatcheev & Shultz (2013) further analyse the model described in (Hammond & Axelrod 2006). The authors have reimplemented the original model and extensively analysed it especially the temporal dynamics. The paper is solely about the original model and its properties, there is no pretence of any validation or calibration with respect to any data. The problem is in the language used, because it the language could equally well refer to the model and the real world.

Take the first sentence of its abstract: “Recent agent-based computer simulations suggest that ethnocentrism, often thought to rely on complex social cognition and learning, may have arisen through biological evolution“. This sounds like the simulation suggests something about the world we live in – that, as the title suggests, Ethnocentric cooperation naturally dominates other strategies (e.g. humanitarianism) and so it is natural. The rest of the abstract then goes on in the same sort of language which could equally apply to the model and the real world.

Expert modellers will understand that they were talking about the purely abstract properties of the model, but this will not be clear to other readers. However, in this case there is evidence that it is a problem. This paper has, in recent years, shot to the top of page requests from the JASSS website (22nd May 2020) at 162,469 requests over a 7-day period, but is nowhere in the top 50 articles in terms of JASSS-JASSS citations. Tracing where these requests come from, results in many alt-right and Russian web sites. It seems that many on the far right see this paper as confirmation of their Nationalist and Racist viewpoints. This is far more attention than a technical paper just about a model would get, so presumably they took it as confirmation about real-world conclusions (or were using it to fool others about the scientific support for their viewpoints) – namely that Ethnocentrism does beat Humanitarianism and this is an evolutionary inevitability [note 1].

This is an extreme example of the confusion that occurs when non-expert modellers read many papers on modelling. Modellers too often imply a degree of real-world relevance when this is not justified by their research. They often imply real-world conclusions before any meaningful validation has been done. As agent-based simulation reaches a less specialised audience, this will become more important.

Some suggestions to avoid this kind of confusion:

  • After the motivation section, carefully outline what part this research will play in the broader programme – do not leave this implicit or imply a larger role than is justified
  • Add in the phrase “in the model” frequently in the text, even if this is a bit repetitive [note 2]
  • Keep  discussions about the real world in a different sections from those that discuss the model
  • Have an explicit statement of what the model can reliably say about the real world
  • Use different terms when referring to parts of the model and part of the real world (e.g. actors for real world individuals, agents in the model)
  • Be clear about the intended purpose of the model – what can be achieved as a result of this research (Edmonds et al. 2019) – for example, do not imply the model will be able to predict future real world properties until this has been demonstrated (de Matos Fernandes & Keijzer 2020)
  • Be very cautious in what you conclude from your model – make sure this is what has been already achieved rather than a reflection of your aspirations (in fact it might be better to not mention such hopes at all until they are realised)

Notes

  1. To see that this kind of conclusion is not necessary see (Hales & Edmonds 2019).
  2. This is similar to a campaign to add the words “in mice” in reports about medical “breakthroughs”, (https://www.statnews.com/2019/04/15/in-mice-twitter-account-hype-science-reporting)

References

Edmonds, B., et al. (2019) Different Modelling Purposes, Journal of Artificial Societies and Social Simulation 22(3), 6. <http://jasss.soc.surrey.ac.uk/22/3/6.html>. doi:10.18564/jasss.3993

Hammond, R. A., N. D. and Axelrod, R. (2006). The Evolution of Ethnocentrism. Journal of Conflict Resolution, 50(6), 926–936. doi:10.1177/0022002706293470

Hartshorn, Max, Kaznatcheev, Artem and Shultz, Thomas (2013) The Evolutionary Dominance of Ethnocentric Cooperation, Journal of Artificial Societies and Social Simulation 16(3), 7. <http://jasss.soc.surrey.ac.uk/16/3/7.html>. doi:10.18564/jasss.2176

Hesse, M. (1964). Analogy and confirmation theory. Philosophy of Science, 31(4), 319-327.

Kuhn, T. S. (1962). The Structure of Scientific Revolutions. Univ. of Chicago Press.

de Matos Fernandes, C. A. and Keijzer, M. A. (2020) No one can predict the future: More than a semantic dispute. Review of Artificial Societies and Social Simulation, 15th April 2020. https://rofasss.org/2020/04/15/no-one-can-predict-the-future/

Wartofsky, M. (1966). the Model Muddle – Proposals for an Immodest Realism. Journal Of Philosophy, 63(19), 589-589.


Edmonds, B. (2020) Basic Modelling Hygiene - keep descriptions about models and what they model clearly distinct. Review of Artificial Societies and Social Simulation, 22nd May 2020. https://rofasss.org/2020/05/22/modelling-hygiene/


 

The Policy Context of Covid19 Agent-Based Modelling

By Edmund Chattoe-Brown

(A contribution to the: JASSS-Covid19-Thread)

In the recent discussions about the role of ABM and COVID, there seems to be an emphasis on the purely technical dimensions of modelling. This obviously involves us “playing to our strengths” but unfortunately it may reduce the effectiveness that our potential policy contributions can make. Here are three contextual aspects of policy for consideration to provide a contrast/corrective.

What is “Good” Policy?

Obviously from a modelling perspective good policy involves achieving stated goals. So a model that suggests a lower death rate (or less taxing of critical care facilities) under one intervention rather than another is a potential argument for that intervention. (Though of course how forceful the argument is depends on the quality of the model.) But the problem is that policy is predominantly a political and not a technical process (related arguments are made by Edmonds 2020). The actual goals by which a policy is evaluated may not be limited to the obvious technical ones (even if that is what we hear most about in the public sphere) and, most problematically, there may be goals which policy makers are unwilling to disclose. Since we do not know what these goals are, we cannot tell whether their ends are legitimate (having to negotiate privately with the powerful to achieve anything) or less so (getting re-elected as an end in itself).

Of course, by its nature (being based on both power and secrecy), this problem may be unfixable but even awareness of it may change our modelling perspective in useful ways. Firstly, when academic advice is accused of irrelevance, the academics can only ever be partly to blame. You can only design good policy to the extent that the policy maker is willing to tell you the full evaluation function (to the extent that they know it of course). Obviously, if policy is being measured by things you can’t know about, your advice is at risk of being of limited value. Secondly, with this is mind, we may be able to gain some insight into the hidden agenda of policy by looking at what kind of suggestions tend to be accepted and rejected. Thirdly, once we recognise that there may be “unknown unknowns” we can start to conjecture intelligently about what these could be and take some account of them in our modelling strategies. For example, how many epidemic models consider the financial costs of interventions even approximately? Is the idea that we can and will afford whatever it takes to reduce deaths a blind spot of the “medical model?”

When and How to Intervene

There used to be an (actually rather odd) saying: “You can’t get a baby in a month by making nine women pregnant”. There has been a huge upsurge in interest regarding modelling and its relationship to policy since start of the COVID crisis (of which this theme is just one example) but realising the value of this interest currently faces significant practical problems. Data collection is even harder than usual (as is scholarship in general), there is a limit to how fast good research can ever be done, peer review takes time and so on. The question here is whether any amount of rushing around at the present moment will compensate for neglected activities when scholarship was easier and had more time (an argument also supported by Bithell 2018). The classic example is the muttering in the ABM community about the Ferguson model being many thousands of lines of undocumented C code. Now we are in a crisis, even making the model available was a big ask, let alone making it easier to read so that people might “heckle” it. But what stopped it being available, documented, externally validated and so on before COVID? What do we need to do so that next time there is a pandemic crisis, which there surely will be, “we” (the modelling community very broadly defined) are able to offer the government a “ready” model that has the best features of various modelling techniques, evidence of unfudgeable quality against data, relevant policy scenarios and so on? (Specifically, how will ABM make sure it deserves to play a fit part in this effort?) Apart from the models themselves, what infrastructures, modelling practices, publishing requirements and so on do we need to set up and get working well while we have the time? In practice, given the challenges of making effective contributions right now (and the proliferation of research that has been made available without time for peer review may be actively harmful), this perspective may be the most important thing we can realistically carry into the “post lockdown” world.

What Happens Afterwards?

ABM has taken such a long time to “get to” policy based on data that looking further than the giving of such advice simply seems to have been beyond us. But since policy is what actually happens, we have a serious problem with counterfactuals. If the government decides to “flatten the curve” rather than seek “herd immunity” then we know how the policy implemented relates to the model “findings” (for good or ill) but not how the policy that was not implemented does. Perhaps the outturn of the policy that looked worse in the model would actually have been better had it been implemented?

Unfortunately (this is not a typo), we are about to have an unprecedently large social data set of comparative experiments in the nature and timing of epidemiological interventions, but ABM needs to be ready and willing to engage with this data. I think that ABM probably has a unique contribution to make in “endogenising” the effects of policy implementation and compliance (rather than seeing these, from a “model fitting” perspective, as structural changes to parameter values) but to make this work, we need to show much more interest in data than we have to date.

In 1971, Dutton and Starbuck, in a worryingly neglected article (cited only once in JASSS since 1998 and even then not in respect of model empirics) reported that 81% of the models they surveyed up to 1969 could not achieve even qualitative measurement in both calibration and validation (with only 4% achieving quantitative measurement in both). As a very rough comparison (but still the best available), Angus and Hassani-Mahmooei (2015) showed that just 13% of articles in JASSS published between 2010 and 2012 displayed “results elements” both from the simulation and using empirical material (but the reader cannot tell whether these are qualitative or quantitative elements or whether their joint presence involves comparison as ABM methodology would indicate). It would be hard to make the case that the situation in respect to ABM and data has therefore improved significantly in 4 decades and it is at least possible that it has got worse!

For the purposes of policy making (in the light of the comments above), what matters of course is not whether the ABM community believes that models without data continue to make a useful contribution but whether policy makers do.

References

Angus, S. D. and Hassani-Mahmooei, B. (2015) “Anarchy” Reigns: A Quantitative Analysis of Agent-Based Modelling Publication Practices in JASSS, 2001-2012, Journal of Artificial Societies and Social Simulation, 18(4), 16. doi:10.18564/jasss.2952

Bithell, M. (2018) Continuous model development: a plea for persistent virtual worlds, Review of Artificial Societies and Social Simulation, 22nd August 2018. https://rofasss.org/2018/08/22/mb

Dutton, John M. and Starbuck, William H. (1971) Computer Simulation Models of Human Behavior: A History of an Intellectual Technology. IEEE Transactions on Systems, Man, and Cybernetics, SMC-1(2), 128–171. doi:10.1109/tsmc.1971.4308269

Edmonds, B. (2020) What more is needed for truly democratically accountable modelling? Review of Artificial Societies and Social Simulation, 2nd May 2020. https://rofasss.org/2020/05/02/democratically-accountable-modelling/


Chattoe-Brown, E. (2020) The Policy Context of Covid19 Agent-Based Modelling. Review of Artificial Societies and Social Simulation, 4th May 2020. https://rofasss.org/2020/05/04/policy-context/


 

What more is needed for Democratically Accountable Modelling?

By Bruce Edmonds

(A contribution to the: JASSS-Covid19-Thread)

In the context of the Covid19 outbreak, the (Squazzoni et al 2020) paper argued for the importance of making complex simulation models open (a call reiterated in Barton et al 2020) and that relevant data needs to be made available to modellers. These are important steps but, I argue, more is needed.

The Central Dilemma

The crux of the dilemma is as follows. Complex and urgent situations (such as the Covid19 pandemic) are beyond the human mind to encompass – there are just too many possible interactions and complexities. For this reason one needs complex models, to leverage some understanding of the situation as a guide for what to do. We can not directly understand the situation, but we can understand some of what a complex model tells us about the situation. The difficulty is that such models are, themselves, complex and difficult to understand. It is easy to deceive oneself using such a model. Professional modellers only just manage to get some understanding of such models (and then, usually, only with help and critique from many other modellers and having worked on it for some time: Edmonds 2020) – politicians and the public have no chance of doing so. Given this situation, any decision-makers or policy actors are in an invidious position – whether to trust what the expert modellers say if it contradicts their own judgement. They will be criticised either way if, in hindsight, that decision appears to have been wrong. Even if the advice supports their judgement there is the danger of giving false confidence.

What options does such a policy maker have? In authoritarian or secretive states there is no problem (for the policy makers) – they can listen to who they like (hiring or firing advisers until they get advice they are satisfied with), and then either claim credit if it turned out to be right or blame the advisers if it was not. However, such decisions are very often not value-free technocratic decisions, but ones that involve complex trade-offs that affect people’s lives. In these cases the democratic process is important for getting good (or at least accountable) decisions. However, democratic debate and scientific rigour often do not mix well [note 1].

A Cautionary Tale

As discussed in (Adoha & Edmonds 2019) Scientific modelling can make things worse, as in the case of the North Atlantic Cod Fisheries Collapse. In this case, the modellers became enmeshed within the standards and wishes of those managing the situation and ended up confirming their wishful thinking. An effect of technocratising the decision-making about how much it is safe to catch had the effect of narrowing down the debate to particular measurement and modelling processes (which turned out to be gravely mistaken). In doing so the modellers contributed to the collapse of the industry, with severe social and ecological consequences.

What to do?

How to best interface between scientific and policy processes is not clear, however some directions are becoming apparent.

  • That the process of developing and giving advice to policy actors should become more transparent, including who is giving advice and on what basis. In particular, any reservations or caveats that the experts add should be open to scrutiny so the line between advice (by the experts) and decision-making (by the politicians) is clearer.
  • That such experts are careful not to over-state or hype their own results. For example, implying that their model can predict (or forecast) the future of complex situations and so anticipate the effects of policy before implementation (de Matos Fernandes and Keijzer 2020). Often a reliable assessment of results only occurs after a period of academic scrutiny and debate.
  • Policy actors need to learn a little bit about modelling, in particular when and how modelling can be reliably used. This is discussed in (Government Office for Science 2018, Calder et al. 2018) which also includes a very useful checklist for policy actors who deal with modellers.
  • That the public learn some maturity about the uncertainties in scientific debate and conclusions. Preliminary results and critiques tend to be jumped on too early to support one side within polarised debate or models rejected simply on the grounds they are not 100% certain. We need to collectively develop ways of facing and living with uncertainty.
  • That the decision-making process is kept as open to input as possible. That the modelling (and its limitations) should not be used as an excuse to limit what the voices that are heard, or the debate to a purely technical one, excluding values (Aodha & Edmonds 2017).
  • That public funding bodies and journals should insist on researchers making their full code and documentation available to others for scrutiny, checking and further development (readers can help by signing the Open Modelling Foundation’s open letter and the campaign for Democratically Accountable Modelling’s manifesto).

Some Relevant Resources

  • CoMSeS.net — a collection of resources for computational model-based science, including a platform for publicly sharing simulation model code and documentation and forums for discussion of relevant issues (including one for covid19 models)
  • The Open Modelling Foundation — an international open science community that works to enable the next generation modelling of human and natural systems, including its standards and methodology.
  • The European Social Simulation Association — which is planning to launch some initiatives to encourage better modelling standards and facilitate access to data.
  • The Campaign for Democratic Modelling — which campaigns concerning the issues described in this article.

Notes

note1: As an example of this see accounts of the relationship between the UK scientific advisory committees and the Government in the Financial Times and BuzzFeed.

References

Barton et al. (2020) Call for transparency of COVID-19 models. Science, Vol. 368(6490), 482-483. doi:10.1126/science.abb8637

Aodha, L.Edmonds, B. (2017) Some pitfalls to beware when applying models to issues of policy relevance. In Edmonds, B. & Meyer, R. (eds.) Simulating Social Complexity – a handbook, 2nd edition. Springer, 801-822. (see also http://cfpm.org/discussionpapers/236)

Calder, M., Craig, C., Culley, D., de Cani, R., Donnelly, C.A., Douglas, R., Edmonds, B., Gascoigne, J., Gilbert, N. Hargrove, C., Hinds, D., Lane, D.C., Mitchell, D., Pavey, G., Robertson, D., Rosewell, B., Sherwin, S., Walport, M. & Wilson, A. (2018) Computational modelling for decision-making: where, why, what, who and how. Royal Society Open Science,

Edmonds, B. (2020) Good Modelling Takes a Lot of Time and Many Eyes. Review of Artificial Societies and Social Simulation, 13th April 2020. https://rofasss.org/2020/04/13/a-lot-of-time-and-many-eyes/

de Matos Fernandes, C. A. and Keijzer, M. A. (2020) No one can predict the future: More than a semantic dispute. Review of Artificial Societies and Social Simulation, 15th April 2020. https://rofasss.org/2020/04/15/no-one-can-predict-the-future/

Government Office for Science (2018) Computational Modelling: Technological Futures. https://www.gov.uk/government/publications/computational-modelling-blackett-review

Squazzoni, F., Polhill, J. G., Edmonds, B., Ahrweiler, P., Antosz, P., Scholz, G., Chappin, É., Borit, M., Verhagen, H., Giardini, F. and Gilbert, N. (2020) Computational Models That Matter During a Global Pandemic Outbreak: A Call to Action. Journal of Artificial Societies and Social Simulation, 23(2):10. <http://jasss.soc.surrey.ac.uk/23/2/10.html>. doi: 10.18564/jasss.4298


Edmonds, B. (2020) What more is needed for truly democratically accountable modelling? Review of Artificial Societies and Social Simulation, 2nd May 2020. https://rofasss.org/2020/05/02/democratically-accountable-modelling/


 

Understanding the current COVID-19 epidemic: one question, one model

By the CoVprehension Collective

(A contribution to the: JASSS-Covid19-Thread)

On the evening of 16th March 2020, the French president, Emmanuel Macron announced the start of a national lockdown, for a period of 15 days. It would be effective from noon the next day (17th March). On the 18th March 2020 at 01:11 pm, the first email circulated in the MicMac team, who had been working on the micro-macro modelling of the spread of a disease in a transportation network a few years. This email was the start of CoVprehension. After about a week of intense emulation, the website was launched, with three questions answered. A month later, there were about fifteen questions on the website, and the group was composed of nearly thirty members from French research institutions, in a varied pool of disciplines, all contributing as volunteers from their confined residence.

CoVprehension in principles

This rapid dynamic originates from a very singular context. It is tricky to analyse it given that the COVID-19 crisis is still developing. However, we can highlight a few fundamental principles leading the project.

The first principle is undeniably a principle of action. To become an actor of the situation first, but this invitation extends to readers of the website, allowing them to run the simulation and to change its parameters; but also more broadly by giving them suggestions on how to link their actions to this global phenomenon which is hard to comprehend. This empowerment also touches upon principles of social justice and, longer term, democracy in the face of this health crisis. By accompanying the process of social awareness, we aim to guide the audience towards a free and informed consent (cf. code of public health) in order to confront the disease. Our first principle is spelled out on theCoVprehension website in the form of a list of objectives that the CoVprehension collective set themselves:

  • Comprehension (the propagation of the virus, the actions put in place)
  • Objectification (giving a more concrete shape to this event which is bigger than us and can be overwhelming)
  • Visualisation (showing the mechanisms at play)
  • Identification (the essential principles and actions to put in place)
  • Do something (overcoming fears and anxieties to become actors in the epidemic)

The second founding principle is that of an interdisciplinary scientific collective formed on a voluntary basis. CoVprehension is self-organised and rests on three pillars: volunteering, collaborative work and the will to be useful during the crisis by offering a space for information, reflection and interaction with a large audience.

As a third principle, we have agility and reactivity. The main idea of the project is to answer questions that people ask, with short posts based on a model or data analysis. This can only be done if the delay between question and answer remains short, which is a real challenge given the complexity of the subject, the high frequency of scientific literature being produced since the beginning of the crisis, and the large number of unknowns and uncertainties which characterise it.

The fourth principle, finally, is the autonomy of groups which form to answer the questions. This allows a multiplicity of perspectives and points of view, sometimes divergent. This necessity draws on the acknowledgement by the European simulation community that a lack of pluralism is even more harmful to support public decision-making than a lack of transparency.

A collaborative organisation and an interactive website

The four principles have lead us, quite naturally, to favour a functioning organisation which exploits short and frequent retroactions and relies of adapted tools. The questions asked online through a Framasoft form are transferred to all CoVprehension members, while a moderator is in charge of replying to them quickly and personally. Each question is integrated into a Trello management board, which allows each member of the collective to pick the questions they want to contribute to and to follow their progression until publication. The collaboration and debate on each of the questions is done using VoIP application Discord. Model prototypes are mostly developed on the Netlogo platform (with some javascript exceptions). Finally, the whole project and website is hosted on GitHub.

The website itself (https://covprehension.org/en) is freely accessible online. Besides the posts answering questions, it contains a simulator to rerun and reproduce the simulations showcased in the posts, a page with scientific resources on the COVID-19 epidemic, a page presenting the project members and a link to the form allowing anyone to ask the collective a question.

On the 28th April 2020, the collective counted 29 members (including 10 women): medical doctors, researchers, engineers and specialists in the fields of computer science, geography, epidemiology, mathematics, economy, data analysis, medicine, architecture and digital media production. The professional statuses of the team members vary (from PhD student to full professor, from intern to engineer, from lecturer to freelancer) whereas their skills complement each other (although a majority of them are complex system modellers). The collective effort enables CoVprehension to scale up on information collection, sharing and updating. This is also fueled by debates during the first take on questions by small teams. Such scaling up would otherwise only be possible in large epidemiology laboratories with massive funding. To increase visibility, the content of the website, initially all in French, is being translated into English progressively as new questions are published.

Simple simulation models

When a question requires a model, especially so for the first questions, our choice has been to build simple models (cf. Question 0). Indeed, the objective of CoVprehension models is not to predict. It is rather to describe, to explain and to illustrate some aspects of the COVID-19 epidemic and its consequences on population. KISS models (“Keep It Simple, Stupid!” cf. Edmonds  & Moss 2004) for the opposition between simple and “descriptive” models) seem better suited to our project. They can unveil broad tendencies and help develop intuitions about potential strategies to deal with the crisis, which can then be also shared with a broad audience.

By choosing a KISS posture, we implicitly reject KIDS postures in such crisis circumstances. Indeed, if the conditions and processes modelled were better informed and known, we could simulate a precise dynamic and generate a series of predictions and forecasts. This is what N. Ferguson’s team did for instance, with a model initially developed with regards to the H5N1 flu in Asia (Ferguson et al., 2005). This model was used heavily to inform public decision-making in the first days of the epidemic in the United Kingdom. Building and calibrating such models takes an awfully long time (Ferguson’s project dates back from 2005) and requires teams and recurring funding which is almost impossible to get nowadays for most teams. At the moment, we think that uncertainty is too big, and that the crisis and the questions that people have do not always necessitate the modelling of complex processes. A large area of the space of social questions mobilised can be answered without describing the mechanisms in so much detail. It is possible that this situation will change as we get information from other scientific disciplines. For now, demonstrating that even simple models are very sensitive to many elements which remain uncertain shows that the scientific discourse could gain by remaining humble: the website reveals how little we know about the future consequences of the epidemic and the political decisions made to tackle it.

Feedback on the questions received and answered

At the end of April, twenty-seven questions have been asked to the CoVprehension collective, through the online form. Seven of them are not really questions (they are rather remarks and comments from people supporting the initiative). Some questions happen to have been asked by colleagues and relatives. The intended outreach has not been fully realised since the website seems to reach people who are already capable of looking for information on the internet. This was to be expected given the circumstances. Everyone who has done some scientific outreach knows how hard it is to reach populations who have not been been made aware of or are interested in scientific facts in the first place. Some successful initiatives (like “les petits débrouillards” or “la main à la pâte” in France) spread scientific knowledge related to recent publications in collaboration with researchers, but they are much better equipped for that (since they do not rely mostly on institutional portals like we do). This large selection bias in our audience (almost impossible to solve, unless we create some specific buzz… which we will then have to handle in terms of new question influx, which is not possible at the moment given the size of the collective and its organisation) means that our website has been protected from trolling. However, we can expect that it might be used within educational programs for example, where STEM teachers could make the students use the various simulators in a question and answer type of game.

Figure 1 shows that the majority of questions are taken by small interdisciplinary teams of two or three members. The most frequent collaborations are between geographers and computer scientists. They are often joined by epidemiologists and mathematicians, and recently by economists. Most topics require the team to build and analyse a simulation model in order to answer the question. The timing of team formations reflects the arrival of new team members in the early days of the project, leading to a large number of questions to be tackled simultaneously. Since April, the rhythm has slowed, reflecting also the increasing complexity of questions, models and answers, but also the marginal “cost” of this investment on the other projects and responsibilities of the researchers involved.

Visualisation of the questions tackled by Covprehension.

Figure 1. Visualisation of the questions tackled by Covprehension.

Initially, the website prioritised questions on simulation and aggregation effects specifically connected with the distribution models of diffusion. For instance, the first questions aimed essentially at showing the most tautological results: with simple interaction rules, we illustrated logically expected effects. These results are nevertheless interesting because while they are trivial to simulation practitioners, they also serve to convince profane readers that they are able to follow the logic:

  • Reducing the density of interactions reduces the spread of the virus and therefore: maybe the lockdown can alter the infection curve (cf. Question 2 and Question 3).
  • By simply adding a variable for the number of hospital beds, we can visualise the impact of lockdown on hospital congestion (cf. Question 7).

For more elaborate questions to be tackled (and to rationalise the debates):

  • Some alternative policies have been highlighted (the Swedish case: Question 13; the deconfinement: Question 9);
  • Some indicators with contradicting impacts have been discussed, which shows the complexity of political decisions and leads readers to question the relevance of some of these indicators (cf. Question 6);
  • The hypotheses (behavioural ones in particular) have been largely discussed, which highlights the way in which the model deviates from what it represents in a simplified way (cf. Question 15).

More than half of the questions asked could not be answered through modelling. In the first phase of the project, we personnally replied to these questions and directed the person towards robust scientific websites or articles where their question could be better answered. The current evolution of the project is more fundamental: new researchers from complementary disciplines have shown some interest in the work done so far and are now integrated into the team (including two medical doctors operating in COVID-19 centres for instance). This will broaden the scope of questions tackled by the team from now on.

Our work fits into a type of education to critical thinking about formal models, one that has long been known as necessary to a technical democracy (Stengers, 2017). At this point, the website can be considered both as a result by itself and as a pilot to function as a model for further initiatives.

Conclusion

Feedback on the CoVprehension project has mostly been positive, but not exempt from limits and weaknesses. Firstly, the necessity of a prompt response has been detrimental to our capacity to fully explore different models, to evaluate their robustness and look for unexpected results. Model validation is unglamorous, slow and hard to communicate. It is crucial nevertheless when assessing the credibility to be associated with models and results. We are now trying to explore our models in parallel. Secondly, the website may suggest a homogeneity of perspectives and a lack of debates regarding how questions are to be answered. These debates do take place during the assessment of questions but so far remain hidden from the readers. It shows indirectly in the way some themes appear in different answers treated from different angles by different teams (for example: the lockdown, treated in question 6, 7, 9 and 14). We consider the possibility of publishing alternative answers to a given question in order to show this possible divergence. Finally, the project is facing a significant challenge: that of continuing its existence in parallel with its members’ activities, with the number of members increasing. The efforts in management, research, editing, publishing and translation have to be maintained while the transaction costs are going up as the size and diversity of the collective increases, as the debates become more and more specific and happen on different platforms… and while new questions keep arriving!

References

Edmonds, B., & Moss, S. (2004). From KISS to KIDS–an ‘anti-simplistic’ modelling approach. In International workshop on multi-agent systems and agent-based simulation (pp. 130-144). Springer, Berlin, Heidelberg. doi:10.1007/978-3-540-32243-6_11

Ferguson, N. M., Cummings, D. A., Cauchemez, S., Fraser, C., Riley, S., Meeyai, A. & Burke, D. S. (2005). Strategies for containing an emerging influenza pandemic in Southeast Asia. Nature, 437(7056), 209-214. doi:10.1038/nature04017

Stengers I. (2017). Civiliser la modernité ? Whitehead et les ruminations du sens commun, Dijon, Les presses du réel. https://www.lespressesdureel.com/EN/ouvrage.php?id=3497


the CoVprehension Collective (2020) Understanding the current COVID-19 epidemic: one question, one model. Review of Artificial Societies and Social Simulation, 30th April 2020. https://rofasss.org/2020/04/30/covprehension/


 

What can and cannot be feasibly modelled of the Covid-19 Pandemic

By Nick Gotts

(A contribution to the: JASSS-Covid19-Thread)

The place of modelling in informing policy has been highlighted by the Covid-19 pandemic. In the UK, a specific individual-based epidemiological model, that developed by Neil Ferguson of Imperial College London, has been credited with the government’s U-turn from pursuing a policy of building up “herd immunity” by allowing the Sars-CoV-2 virus to spread through the population in order to avoid a possible “second wave” next winter (while trying to limit the speed of spread so as to avoid overwhelming medical facilities, and to shield the most vulnerable), to a “lockdown” imposed in order to minimise the number of people infected. Ferguson’s model reportedly indicated several hundred thousand deaths if the original policy was followed, and this was judged unacceptable.

I do not doubt that the reversal of policy was correct – indeed, that the original policy should never have been considered – one prominent epidemiologist said he thought the report of it was “satire” when he first heard it (Hanage 2020). As Hanage says: “Vulnerable people should not be exposed to Covid-19 right now in the service of a hypothetical future”. But it has also been reported (Reynolds 2020) that Ferguson’s model is a rapid modification of one he built to study possible policy responses to a hypothetical influenza pandemic (Ferguson et al. 2006); and that (Ferguson himself says) this model consists of “thousands of lines of undocumented C”. That major policy decisions should be made on such a basis is both wrong in itself, and threatens to bring scientific modelling into disrepute – indeed, I have already seen the justified questioning of the UK government’s reliance on modelling used by climate change denialists in their ceaseless quest to attack climate science.

What can social simulation contribute in the Covid-19 crisis? I suggest that attempts to model the pandemic as a whole, or even in individual countries, are fundamentally misplaced at this stage: too little is known about the behaviour of the virus, and governments need to take decisions on a timescale that simply does not allow for responsible modelling practice. Where social simulation might be of immediate use is in relation to the local application of policies already decided on. To give one example, supermarkets in the UK (and I assume, elsewhere) are now limiting the number of shoppers in their stores at any one time, in an effort to apply the guidelines on maintaining physical distance between individuals from different households. But how many people should be permitted in a given store? Experience from traffic models suggests there may well be a critical point at which it rather suddenly becomes impossible to maintain distance as the number of shoppers increases – but where does it lie for a particular store? Could the goods on sale be rearranged in ways that allow larger numbers – for example, by distributing items in high demand across two or more aisles? Supermarkets collect a lot of information about what is bought, and which items tend to be bought together – could they shorten individual shoppers’ time in the store by improving their signage? (Under normal circumstances, of course, they are likely to want to retain shoppers as long as possible, and send them down as many aisles as possible, to encourage impulse buys.)

Agents in such a model could be assigned a list of desired purchases, speed of movement and of collecting items from shelves, and constraints on how close they come to other shoppers – probably with some individual variation. I would be interested to learn if any modelling teams have approached supermarket chains (or vice versa) with a proposal for such a model, which should be readily adaptable to different stores. Other possibilities include models of how police should be distributed over an area to best ensure they will see (and be seen by) individuals or groups disregarding constraints on gathering in groups, and of the “contagiousness” of such behaviour – which, unlike actual Covid-19 infection events, is readily observable. Social simulators, in summary, should look for things they can reasonably hope to do quickly and in conjunction with organisations that have or can readily collect the required data, not try to do what is way beyond what is possible in the time available.

References

Ferguson, N. M., Cummings, D. A., Fraser, C., Cajka, J. C., Cooley, P. C., & Burke, D. S. (2006). Strategies for mitigating an influenza pandemic. Nature, 442(7101), 448-452. doi:10.1038/nature04795

Hanage, W. (2020) I’m an epidemiologist. When I heard about Britain’s ‘herd immunity’ coronavirus plan, I thought it was satire. The Guardian, 2020-03-15. https://www.theguardian.com/commentisfree/2020/mar/15/epidemiologist-britain-herd-immunity-coronavirus-covid-19

Reynolds, C. (2020) Big Tech Fights Back: From Pandemic Simulation Code, to Immune Response. Computer Business Review 2020-03-15. https://www.cbronline.com/news/pandemic-simulation-code.


Gotts, N. (2020) What can and cannot be feasibly modelled of the Covid-19 Pandemic. Review of Artificial Societies and Social Simulation, 29th April 2020. https://rofasss.org/2020/04/29/feasibility/


 

The Danger of too much Compassion – how modellers can easily deceive themselves

By Andreas Tolk

(A contribution to the: JASSS-Covid19-Thread)

In 2017, Shermer observed that in cases where moral and epistemological considerations are deeply intertwined, it is human nature to cherry-pick the results and data that support the current world view (Shermer 2017). In other words, we tend to look for data justifying our moral conviction. The same is an inherent challenge for simulations as well: we tend to favour our underlying assumptions and biases – often even unconsciously – when we implement our simulation systems. If now others use this simulation system in support of predictive analysis, we are in danger of philosophical regress: a series of statements in which a logical procedure is continually reapplied to its own result without approaching a useful conclusion. As stated in an earlier paper of mine (Tolk 2017):

The danger of the simulationist’s regress is that such predictions are made by the theory, and then the implementation of the theory in form of the simulation system is used to conduct a simulation experiment that is then used as supporting evidence. This, however, is exactly the regress we wanted to avoid: we test a hypothesis by implementing it as a simulation, and then use the simulated data in lieu of empirical data as supporting evidence justifying the propositions: we create a series of statements – the theory, the simulation, and the resulting simulated data – in which a logical procedure is continually reapplied to its own result….

In particular in cases where moral and epistemological considerations are deeply intertwined, it is human nature to cherry-pick the results and data that support the current world view (Shermer 2017). Simulationists are not immune to this, and as they can implement their beliefs into a complex simulation system that now can be used by others to gain quasi-empirical numerical insight into the behavior of the described complex system, their implemented world view can easily be confused with a surrogate for real world experiments.

I am afraid that we may have fallen into such a fallacy in some of our efforts to use simulation to better understand the Covid-19 crisis and what we can do. This is for sure a moral problem, as at the end of our recommendations this is about human lives! And we assumed that the recommendations of the medical community for social distancing and other non pharmaceutical interventions (NPI) is the best we can do, as it saves many lives. So we built our models to clearly demonstrate the benefits of social distancing and other NPIs, which leads to danger of regress: we assume that NPIs are the best action, so we write a simulation to show that NPIs are the best action, and then we use these simulations to prove that NPIs are the best action. But can we actually use empirical data to support these assumptions? Looking closely at the data, the correlation of success – measured as flattening the curves – and the amount and strictness of the NPIs is not always observable. So we may have missed something, as our model-based predictions are not supported as we hope for, which is a problem: do we just collect the wrong data and should use something else to validate the models, or are the models insufficient to explain the data? And how do we ensure that our passion doesn’t interfere with our scientific objectivity?

One way to address this issue is diversity of opinion implemented as a set of orchestrated models, to use a multitude of models instead of just one. In another comment, the idea of using exploratory analysis to support decision making under deep uncertainty is mentioned. I highly recommend to have a look at (Marchau, Bloemen & Popper 2019) Decision Making Under Deep Uncertainty: From Theory to Practice. I am optimistic that if we are inclusive of a diversity of ideas – even if we don’t like them – and allow for computational evaluation of ALL options using exploratory analysis, we may find a way for better supporting the community.

References

Marchau, V. A., Walker, W. E., Bloemen, P. J., & Popper, S. W. (2019). Decision making under deep uncertainty. Springer. doi:10.1007/978-3-030-05252-2

Tolk, A. (2017, April). Bias ex silico: observations on simulationist’s regress. In Proceedings of the 50th Annual Simulation Symposium. Society for Computer Simulation International. ANSS ’17: Proceedings of the 50th Annual Simulation Symposium, April 2017 Article No.: 15 Pages 1–9. https://dl.acm.org/citation.cfm?id=3106403

Shermer, M. (2017) How to Convince Someone When Facts Fail – Why worldview threats undermine evidence. Scientific American, 316, 1, 69 (January 2017). doi:10.1038/scientificamerican0117-69


Tolk, A. (2020) The Danger of too much Compassion - how modellers can easily deceive themselves. Review of Artificial Societies and Social Simulation, 28th April 2020. https://rofasss.org/2020/04/28/self-deception/


 

For discussion about social simulation research