Tag Archives: social simulation

A Bibliography of ABM Research Explicitly Comparing Real and Simulated Data for Validation

By Edmund Chattoe-Brown

The Motivation

Research that confronts models with data is still sufficiently rare that it is hard to get a representative sense of how it is done and how convincing the results are simply by “background reading”. One way to advance good quality empirical modelling is therefore simply to make it more visible in quantity. With this in mind I have constructed (building on the work of Angus and Hassani-Mahmooei 2015) the first version of a bibliography listing all ABM attempting empirical validation in JASSS between 1998 and 2019 (along with a few other example) – which generates 68 items in all. Each entry gives a full reference and also describes what comparisons are made and where in the article they occur. In addition the document contains a provisional bibliography of articles giving advice or technical support to validation and lists three survey articles that categorise large samples of simulations by their relationships to data (which served as actual or potential sources for the bibliography).

With thanks to Bruce Edmonds, this first version of the bibliography has been made available as a Centre for Policy Modelling Discussion Paper CPM-20-216, which can be downloaded http://cfpm.org/discussionpapers/256.

The Argument

It may seem quite surprising to focus only on validation initially but there is an argument (Chattoe-Brown 2019) which says that this is a more fundamental challenge to the quality of a model than calibration. A model that cannot track real data well, even when its parameters are tuned to do so is clearly a fundamentally inadequate model. Only once some measure of validation has been achieved can we decide how “convincing” it is (comparing independent empirical calibration with parameter tuning for example). Arguably, without validation, we cannot really be sure whether a model tells us anything about the real world at all (no matter how plausible any narrative about its assumptions may appear). This can be seen as a consequence of the arguments about complexity routinely made by ABM practitioners as the plausibility of the assumptions does not map intuitively onto the plausibility of the outputs.

The Uses

Although these are covered in the preface to the bibliography in greater detail, such a sample has a number of scientific uses which I hope will form the basis for further research.

  • To identify (and justify) good and bad practice, thus promoting good practice.
  • To identify (and then perhaps fill) gaps in the set of technical tools needed to support validation (for example involving particular sorts of data).
  • To test the feasibility and value of general advice offered on validation to date and refine it in the face of practical challenges faced by analysis of real cases.
  • To allow new models to demonstrably outperform the levels of validation achieved by existing models (thus creating the possibility for progressive empirical research in ABM).
  • To support agreement about the effective use of the term validation and to distinguish it from related concepts (like verification) and potentially unhelpful (for example ambiguous or rhetorically loaded) uses

The Plan

Because of the labour involved and the diversity of fields in which ABM have now been used over several decades, an effective bibliography on this kind cannot be the work of a single author (or even a team of authors). My plan is thus to solicit (fully credited) contributions and regularly release new versions of the bibliography – with new co-authors as appropriate. (This publishing model is intended to maintain the quality and suitability for citation of the resulting document relative to the anarchy that sometimes arises in genuine communal authorship!) All of the following contributions will be gratefully accepted for the next revision (on which I am already working myself in any event)

  • References to new surveys or literature reviews that categorise significant samples of ABM research by their relationship to data.
  • References for proposed new entries to the bibliography in as much detail as possible.
  • Proposals to delete incorrectly categorised entries. (There are a small number of cases where I have found it very difficult to establish exactly what the authors did in the name of validation, partly as a result of confusing or ambiguous terminology.)
  • Proposed revisions to incorrect or “unfair” descriptions of existing entries (ideally by the authors of those pieces).
  • Offers of collaboration for a proposed companion bibliography on calibration. Ultimately this will lead to a (likely very small) sample of calibrated and validated ABM (which are often surprisingly little cited given their importance to the credibility of the ABM “project” – see, for example, Chattoe-Brown (2018a, 2018b).

References

Angus, Simon D. and Hassani-Mahmooei, Behrooz (2015) ‘“Anarchy” Reigns: A Quantitative Analysis of Agent-Based Modelling Publication Practices in JASSS, 2001-2012’, Journal of Artificial Societies and Social Simulation, 18(4), October, article 16. <http://jasss.soc.surrey.ac.uk/18/4/16.html> doi:10.18564/jasss.2952

Chattoe-Brown, Edmund (2018a) ‘Query: What is the Earliest Example of a Social Science Simulation (that is Nonetheless Arguably an ABM) and Shows Real and Simulated Data in the Same Figure or Table?’ Review of Artificial Societies and Social Simulation, 11 June. https://rofasss.org/2018/06/11/ecb/

Chattoe-Brown, Edmund (2018b) ‘A Forgotten Contribution: Jean-Paul Grémy’s Empirically Informed Simulation of Emerging Attitude/Career Choice Congruence (1974)’, Review of Artificial Societies and Social Simulation, 1 June. https://rofasss.org/2018/06/01/ecb/

Chattoe-Brown, Edmund (2019) ‘Agent Based Models’, in Atkinson, Paul, Delamont, Sara, Cernat, Alexandru, Sakshaug, Joseph W. and Williams, Richard A. (eds.) SAGE Research Methods Foundations. doi:10.4135/9781526421036836969


Chattoe-Brown, E. (2020) A Bibliography of ABM Research Explicitly Comparing Real and Simulated Data for Validation. Review of Artificial Societies and Social Simulation, 12th June 2020. https://rofasss.org/2020/06/12/abm-validation-bib/


 

Get out of your silos and work together!

By Peer-Olaf Siebers and Sudhir Venkatesan

(A contribution to the: JASSS-Covid19-Thread)

The JASSS position paper ‘Computational Models That Matter During a Global Pandemic Outbreak: A Call to Action’ (Squazzoni et al 2020) calls on the scientific community to improve the transparency, access, and rigour of their models. A topic that we think is equally important and should be part of this list is the quest to more “interdisciplinarity”; scientific communities to work together to tackle the difficult job of understanding the complex situation we are currently in and be able to give advice.

The modelling/simulation community in the UK (and more broadly) tend to work in silos. The two big communities that we have been exposed to are the epidemiological modelling community, and social simulation community. They do not usually collaborate with each other despite working on very similar problems and using similar methods (e.g. agent-based modelling). They publish in different journals, use different software, attend different conferences, and even sometimes use different terminology to refer to the same concepts.

The UK pandemic response strategy (Gov.UK 2020) is guided by advice from the Scientific Advisory Group for Emergencies (SAGE), which in turn has comprises three independent expert groups- SPI-M (epidemic modellers), SPI-B (experts in behaviour change from psychology, anthropology and history), and NERVTAG (clinicians, epidemiologists, virologists and other experts). Of these, modelling from member SPI-M institutions has played an important role in informing the UK government’s response to the ongoing pandemic (e.g. Ferguson et al 2020). Current members of the SPI-M belong to what could be considered the ‘epidemic modelling community’. Their models tend to be heavily data-dependent which is justifiable given that their most of their modelling focus on viral transmission parameters. However, this emphasis on empirical data can sometimes lead them to not model behaviour change or model it in a highly stylised fashion, although more examples of epidemic-behaviour models appear in recent epidemiological literature (e.g. Verelst et al 2016; Durham et al 2012; van Boven et al 2008; Venkatesan et al 2019). Yet, of the modelling work informing the current response to the ongoing pandemic, computational models of behaviour change are prominently missing. This, from what we have seen, is where the ‘social simulation’ community can really contribute their expertise and modelling methodologies in a very valuable way. A good resource for epidemiologists in finding out more about the wide spectrum of modelling ideas are the Social Simulation Conference Proceeding Programmes (e.g. SSC2019 2019). But unfortunately, the public health community, including policymakers, are either unaware of these modelling ideas or are unsure of how these are relevant to them.

As pointed out in a recent article, one important concern with how behaviour change has possibly been modelled in the SPI-M COVID-19 models is the assumption that changes in contact rates resulting from a lockdown in the UK and the USA will mimic those obtained from surveys performed in China, which unlikely to be valid given the large political and cultural differences between these societies (Adam 2020). For the immediate COVID-19 response models, perhaps requiring cross-disciplinary validation for all models that feed into policy may be a valuable step towards more credible models.

Effective collaboration between academic communities relies on there being a degree of familiarity, and trust, with each other’s work, and much of this will need to be built up during inter-pandemic periods (i.e. “peace time”). In the long term, publishing and presenting in each other’s journals and conferences (i.e. giving the opportunity for other academic communities to peer-review a piece of modelling work), could help foster a more collaborative environment, ensuring that we are in a much better to position to leverage all available expertise during a future emergency. We should aim to take the best across modelling communities and work together to come up with hybrid modelling solutions that provide insight by delivering statistics as well as narratives (Moss 2020). Working in silos is both unhelpful and inefficient.

References

Adam D (2020) Special report: The simulations driving the world’s response to COVID-19. How epidemiologists rushed to model the coronavirus pandemic. Nature – News Feature. https://www.nature.com/articles/d41586-020-01003-6 [last accessed 07/04/2020]

Durham DP, Casman EA (2012) Incorporating individual health-protective decisions into disease transmission models: A mathematical framework. Journal of The Royal Society Interface. 9(68), 562-570

Ferguson N, Laydon D, Nedjati Gilani G, Imai N, Ainslie K, Baguelin M, Bhatia S, Boonyasiri A, Cucunuba Perez Zu, Cuomo-Dannenburg G, Dighe A (2020) Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf [last accessed 07/04/2020]

Gov.UK (2020) Scientific Advisory Group for Emergencies (SAGE): Coronavirus response. https://www.gov.uk/government/groups/scientific-advisory-group-for-emergencies-sage-coronavirus-covid-19-response [last accessed 07/04/2020]

Moss S (2020) “SIMSOC Discussion: How can disease models be made useful? “, Posted by Scott Moss, 22 March 2020 10:26 [last accessed 07/04/2020]

Squazzoni F, Polhill JG, Edmonds B, Ahrweiler P, Antosz P, Scholz G, Borit M, Verhagen H, Giardini F, Gilbert N (2020) Computational models that matter during a global pandemic outbreak: A call to action, Journal of Artificial Societies and Social Simulation, 23 (2) 10

SSC2019 (2019) Social simulation conference programme 2019. https://ssc2019.uni-mainz.de/files/2019/09/ssc19_final.pdf [last accessed 07/04/2020]

van Boven M, Klinkenberg D, Pen I, Weissing FJ, Heesterbeek H (2008) Self-interest versus group-interest in antiviral control. PLoS One. 3(2)

Venkatesan S, Nguyen-Van-Tam JS, Siebers PO (2019) A novel framework for evaluating the impact of individual decision-making on public health outcomes and its potential application to study antiviral treatment collection during an influenza pandemic. PLoS One. 14(10)

Verelst F, Willem L, Beutels P (2016) Behavioural change models for infectious disease transmission: A systematic review (2010–2015). Journal of The Royal Society Interface. 13(125)


Siebers, P-O. and Venkatesan, S. (2020) Get out of your silos and work together. Review of Artificial Societies and Social Simulation, 8th April 2020. https://rofasss.org/2020/0408/get-out-of-your-silos


 

Call for responses to the JASSS Covid19 position paper

In the recent position paper in JASSS, entitled “Computational Models That Matter During a Global Pandemic Outbreak: A Call to Action” the authors suggest some collective actions we, as social simulators, could take.

We are asking for submissions that present serious comments on this paper. This  could include:

  • To discuss other points of view
  • To talk about possible modelling approaches
  • To review simulation modelling of covid19 that includes social aspects
  • To point out some of the difficulties of interpretation and the interface with the policy/political world
  • To discuss or suggest other possible collective actions that could be taken.

All such contributions will form the the: JASSS-Covid19-Thread


Agent-Based Modelling Pioneers: An Interview with Jim Doran

By David Hales and Jim Doran

Jim Doran is an ABM pioneer. Specifically applying ABM to social phenomena. He has been working on these ideas since the 1960’s. His work made a major contribution to establishing the area as it exists today.

In fact Jim has made significant contributions in many areas related to computation such as Artificial Intelligence (AI), Distributed AI (DAI) and Multi-agent Systems (MAS).

I know Jim — he was my PhD supervisor (at the University of Essex) so I had regular meetings with him over a period of about four years. It is hard to capture both the depth and breadth of Jim’s approach. Basically he thinks big. I mean really big! — yet plausibly and precisely. This is a very difficult trick to pull off. Believe me I’ve tried.

He retired from Essex almost two decades ago but continues to work on a number of very innovative ABM related projects that are discussed in the interview.

The interview was conducted over e-mail in August. We did a couple of iterations and included references to the work mentioned.


According to your webpage, at the University of Essex [1] , your background was originally mathematics and then Artificial Intelligence (working with Donald Michie at Edinburgh). In those days AI was a very new area. I wonder if you could say a little about how you came to work with Michie and what kind of things you worked on?

Whilst reading Mathematics at Oxford, I both joined the University Archaeological Society (inspired by the TV archaeologist of the day, Sir Mortimer Wheeler) becoming a (lowest grade) digger and encountering some real archaeologists like Dennis Britten, David Clarke and Roy Hodson, and also, at postgraduate level, was lucky enough to come under the influence of a forward thinking and quite distinguished biometrist, Norman T. J. Bailey, who at that time was using a small computer (an Elliot 803, I think it was) to simulate epidemics — i.e. a variety of computer simulation of social phenomena (Bailey 1967). One day, Bailey told me of this crazy but energetic Reader at Edinburgh University, Donald Michie, who was trying to program computers to play games and to display AI, and who was recruiting assistants. In due course I got a job as an Research Assistant / Junior Research Fellow in Michie’s group (the EPU, for Experimental Programming Unit). During the war Michie had worked with and had been inspired by Alan Turing (see: Lee and Holtzman 1995) [2].

Given this was the very early days of AI, What was it like working at the EPU at that time? Did you meet any other early AI researchers there?

Well, I remember plenty of energy, plenty of parties and visitors from all over including both the USSR (not easy at that time!) and the USA. The people I was working alongside – notably, but not only, Rod Burstall [3], (the late) Robin Popplestone [4], Andrew Ortony [5] – have typically had very successful academic research careers.

I notice that you wrote a paper with Michie in 1966 “Experiments with the graph traverser program”. Am I right, that this is a very early implementation of a generalised search algorithm?

When I took up the research job in Edinburgh at the EPU, in 1964 I think, Donald Michie introduced me to the work by Arthur Samuel on a learning Checkers playing program (Samuel 1959) and proposed to me that I attempt to use Samuel’s rather successful ideas and heuristics to build a general problem solving program — as a rival to the existing if somewhat ineffective and pretentious Newell, Shaw and Simon GPS (Newell et al 1959). The Graph Traverser was the result – one of the first standardised heuristic search techniques and a significant contribution to the foundations of that branch of AI (Doran and Michie 1966) [6]. It’s relevant to ABM because cognition involves planning and AI planning systems often use heuristic search to create plans that when executed achieve desired goals.

Can you recall when you first became aware of and / or began to think about simulating social phenomena using computational agents?

I guess the answer to your question depends on the definition of “computational agent”. My definition of a “computational agent” (today!) is any locus of slightly human like decision-making or behaviour within a computational process. If there is more than one then we have a multi-agent system.

Given the broad context that brought me to the EPU it was inevitable that I would get to think about what is now called agent based modelling (ABM) of social systems – note that archaeology is all about social systems and their long term dynamics! Thus in my (rag bag!) postgraduate dissertation (1964), I briefly discussed how one might simulate on a computer the dynamics of the set of types of pottery (say) characteristic of a particular culture – thus an ABM of a particular type of social dynamics. By 1975 I was writing a critical review of past mathematical modelling and computer simulation in archaeology with prospects (chapter 11 of Doran and Hodson, 1975).

But I didn’t myself use the word “agent” in a publication until, I believe, 1985 in a chapter I contributed to the little book by Gilbert and Heath (1985). Earlier I tended to use the word “actor” with the same meaning. Of course, once Distributed AI emerged as a branch of AI, ABM too was bound to emerge.

Didn’t you write a paper once titled something like “experiments with a pleasure seeking ant in a grid world”? I ask this speculatively because I have some memory of it but can find no references to it on the web.

Yes. The title you are after is “Experiments with a pleasure seeking automaton” published in the volume Machine Intelligence 3 (edited by Michie from the EPU) in 1968. And there was a follow up paper in Machine Intelligence 4 in 1969 (Doran 1968; 1969). These early papers address the combination of heuristic search with planning, plan execution and action within a computational agent but, as you just remarked, they attracted very little attention.

You make an interesting point about how you, today, define a computational agent. Do you have any thoughts on how one would go about trying to identify “agents” in a computational, or other, process? It seems as humans we do this all the time, but could we formalise it in some way?

Yes. I have already had a go at this, in a very limited way. It really boils down to, given the specification of a complex system, searching thru it for subsystems that have particular properties e.g. that demonstrably have memory within their structure of what has happened to them. This is a matter of finding a consistent relationship between the content of the hypothetical agent’s hypothetical memory and the actual input-output history (within the containing complex system) of that hypothetical agent – but the searches get very large. See, for example, my 2002 paper “Agents and MAS in STaMs” (Doran 2002).

From your experience what would you say are the main benefits and limitations of working with agent-based models of social phenomena?

The great benefit is, I feel, precision – the same benefit that mathematical models bring to science generally – including the precise handling of cognitive factors. The computer supports the derivation of the precise consequences of precise assumptions way beyond the powers of the human brain. A downside is that precision often implies particularisation. One can state easily enough that “cooperation is usually beneficial in complex environments”, but demonstrating the truth or otherwise of this vague thesis in computational terms requires precise specification of “cooperation, “complex” and “environment” and one often ends up trying to prove many different results corresponding to the many different interpretations of the thesis.

You’ve produced a number of works that could be termed “computationally assisted thought experiments”, for example, your work on foreknowledge (Doran 1997) and collective misbelief (1998). What do you think makes for a “good” computational thought experiment?

If an experiment and its results casts light upon the properties of real social systems or of possible social systems (and what social systems are NOT possible?), then that has got to be good if only by adding to our store of currently useless knowledge!

Perhaps I should clarify: I distinguish sharply between human societies (and other natural societies) and computational societies. The latter may be used as models of the former, but can be conceived, created and studied in their own right. If I build a couple of hundred or so learning and intercommunicating robots and let them play around in my back garden, perhaps they will evolve a type of society that has NEVER existed before… Or can it be proved that this is impossible?

The recently reissued classic book “Simulating Societies” (Gilbert and Doran 1994, 2018) contains contributions from several of the early researchers working in the area. Could you say a little about how this group came together?

Well – better to ask Nigel Gilbert this question – he organised the meeting that gave rise to the book, and although it’s quite likely I was involved in the choice of invitees, I have no memory. But note there were two main types of contributor – the mainstream social science oriented and the archaeologically oriented, corresponding to Nigel and myself respectively.

Looking back, what would you say have been the main successes in the area?

So many projects have been completed and are ongoing — I’m not going to try to pick out one or two as particularly successful. But getting the whole idea of social science ABM established and widely accepted as useful or potentially useful (along with AI, of course) is a massive achievement.

Looking forward, what do you think are the main challenges for the area?

There are many but I can give two broad challenges:

(i) Finding out how best to discover what levels of abstraction are both tractable and effective in particular modelling domains. At present I get the impression that the level of abstraction of a model is usually set by whatever seems natural or for which there is precedent – but that is too simple.

(Ii) Stopping the use of AI and social ABM being dominated by military and business applications that benefit only particular interests. I am quite pessimistic about this. It seems all too clear that when the very survival of nations, or totalitarian regimes, or massive global corporations is at stake, ethical and humanitarian restrictions and prohibitions, even those internationally agreed and promulgated by the UN, will likely be ignored. Compare, for example, the recent talk by Cristiano Castelfranchi entitled “For a Science-oriented AI and not Servant of the Business”. (Castelfranchi 2018)

What are you currently thinking about?

Three things. Firstly, my personal retirement project, MoHAT — how best to use AI and ABM to help discover effective methods of achieving much needed global cooperation.

The obvious approach is: collect LOTS of global data, build a theoretically supported and plausible model, try to validate it and then try out different ways of enhancing cooperation. MoHAT, by contrast, emphasises:

(i) Finding a high level of abstraction for modelling which is effective but tractable.

(ii) Finding particular long time span global models by reference to fundamental boundary conditions, not by way of observations at particular times and places. This involves a massive search through possible combinations of basic model elements but computers are good at that — hence AI Heuristic Search is key.

(iii) Trying to overcome the ubiquitous reluctance of global organisational structures, e.g. nation states, fully to cooperate – by exploring, for example what actions leading to enhanced global cooperation, if any, are available to one particular state.

Of course, any form of globalism is currently politically unpopular — MoHAT is swimming against the tide!

Full details of MoHAT (including some simple computer code) are in the corresponding project entry in my Research Gate profile (Doran 2018a).

Secondly, Gillian’s Hoop and how one assesses its plausibility as a “modern” metaphysical theory. Gillian’s Hoop is a somewhat wild speculation that one of my daughters came up with a few years ago: we are all avatars in a virtual world created by game players in a higher world who in fact are themselves avatars in a virtual world created by players in a yet higher world … with the upward chain of virtual worlds ultimately linking back to form a hoop! Think about that!

More generally I conjecture that metaphysical systems (e.g. the Roman Catholicism that I grew up with, Gillian’s Hoop, Iamblichus’ system [7], Homer’s) all emerge from the properties of our thought processes. The individual comes up with generalised beliefs and possibilities (e.g. Homer’s flying chariot) and these are socially propagated, revised and pulled together into coherent belief systems. This is little to do with what is there, much more to do with the processes that modify beliefs. This is not a new idea, of course, but it would be good to ground it in some computational modelling.

Again, there is a project description on Research Gate (Doran 2018b).

Finally, I’m thinking about planning and imagination and their interactions and consequences. I’ve put together a computational version of our basic subjective stream of thoughts (incorporating both directed and associative thinking) that can be used to address imagination and its uses. This is not as difficult to come up with as might at first appear. And then comes a conjecture — given ANY set of beliefs, concepts, memories etc in a particular representation system (cf. AI Knowledge Representation studies) it will be possible to define a (or a few) modification processes that bring about generalisations and imaginations – all needed for planning — which is all about deploying imaginations usefully.

In fact I am tempted to follow my nose and assert that:

Imagination is required for planning (itself required for survival in complex environments) and necessarily leads to “metaphysical” belief systems

Might be a good place to stop – any further and I am really into fantasy land…

Notes

  1. Archived copy of Jim Doran’s University of Essex homepage: https://bit.ly/2Pdk4Nf
  2. Also see an online video of some of the interviews, including with Michie, used as a source for the Lee and Holtzman paper: https://youtu.be/6p3mhkNgRXs
  3. https://en.wikipedia.org/wiki/Rod_Burstall
  4. https://en.wikipedia.org/wiki/Robin_Popplestone
  5. https://www.researchgate.net/profile/Andrew_Orton
  6. See also discussion of the historical context of the Graph Traverser in Russell and Norvig (1995).
  7. https://en.wikipedia.org/wiki/Iamblichus

References

Bailey, Norman T. J. (1967) The simulation of stochastic epidemics in two dimensions. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 4: Biology and Problems of Health, 237–257, University of California Press, Berkeley, Calif. https://bit.ly/2or7sqp

Castelfranchi, C. (2018) For a Science-oriented AI and not Servant of the Business. Powerpoint file available from the author on request at Research Gate: https://www.researchgate.net/profile/Cristiano_Castelfranchi

Doran, J.E and Michie, D. (1966) Experiments with the Graph Traverser Program. September 1966. Proceedings of The Royal Society A 294(1437):235-259.

Doran, J.E. (1968) Experiments with a pleasure seeking automaton. In Machine Intelligence 3 (ed. D. Michie) Edinburgh University Press, pp 195-216.

Doran, J.E. (1969) Planning and generalization in an automaton-environment system. In Machine Intelligence 4 (eds. B. Meltzer and D. Michie) Edinburgh University Press. pp 433-454.

Doran, J.E and Hodson, F.R (1975) Mathematics and Computers in Archaeology. Edinburgh University Press, 1975 [and Harvard University Press, 1976]

Doran, J.E. (1997) Foreknowledge in Artificial Societies. In: Conte R., Hegselmann R., Terna P. (eds) Simulating Social Phenomena. Lecture Notes in Economics and Mathematical Systems, vol 456. Springer, Berlin, Heidelberg. https://bit.ly/2Pf5Onv

Doran, J.E. (1998) Simulating Collective Misbelief. Journal of Artificial Societies and Social Simulation vol. 1, no. 1, http://jasss.soc.surrey.ac.uk/1/1/3.html

Doran, J.E. (2002) Agents and MAS in STaMs. In Foundations and Applications of Multi-Agent Systems: UKMAS Workshop 1996-2000, Selected Papers (eds. M d’Inverno, M Luck, M Fisher, C Preist), Springer Verlag, LNCS 2403, July 2002, pp. 131-151. https://bit.ly/2wsrHYG

Doran, J.E. (2018a) MoHAT — a new AI heuristic search based method of DISCOVERING and USING tractable and reliable agent-based computational models of human society. Research Gate Project: https://bit.ly/2lST35a

Doran, J.E. (2018b) An Investigation of Gillian’s HOOP: a speculation in computer games, virtual reality and METAPHYSICS. Research Gate Project: https://bit.ly/2C990zn

Gilbert, N. and Doran, J.E. eds. (2018) Simulating Societies: The Computer Simulation of Social Phenomena. Routledge Library Editions: Artificial Intelligence, Vol 6, Routledge: London and New York.

Gilbert, N. and Heath, C. (1985) Social Action and Artificial Intelligence. London: Gower.

Lee, J. and Holtzman, G. (1995) 50 Years after breaking the codes: interviews with two of the Bletchley Park scientists. IEEE Annals of the History of Computing, vol. 17, no. 1, pp. 32-43. https://ieeexplore.ieee.org/document/366512/

Newell, A.; Shaw, J.C.; Simon, H.A. (1959) Report on a general problem-solving program. Proceedings of the International Conference on Information Processing. pp. 256–264.

Russell, S. and Norvig, P. (1995) Artificial Intelligence: A Modern Approach. Prentice-Hall, First edition, pp. 86, 115-117.

Samuel, Arthur L. (1959) “Some Studies in Machine Learning Using the Game of Checkers”. IBM Journal of Research and Development. doi:10.1147/rd.441.0206.


Hales, D. and Doran, J, (2018) Agent-Based Modelling Pioneers: An Interview with Jim Doran, Review of Artificial Societies and Social Simulation, 4th September 2018. https://rofasss.org/2018/09/04/dh/