In November 2021 Chattoe-Brown initiated a discussion at the SimSoc list on validation which generated quite some traffic on the list. The interest in this topic revealed that empirical validation provides a notorious challenge for agent-based modelling. The discussion raised many important points and questions which even motivated a “smalltalk about big things” at the Social Simulation Fest 2022. Many contributors highlighted that validation cannot be reduced to the comparison of numbers between simulated and empirical data. Without attempting a comprehensive review of this insightful discussion, it has been emphasized that different kinds of science call for different kinds of quality criteria. Prediction might be one criterium that is particularly important in statistics, but that is not sufficient for agent-based social simulation. For instance, agent-based modelling is specifically suited for studying complex systems and turbulent phenomena. Modelling also enables studying alternative and counterfactual scenarios which deviates from the paradigm of prediction as quality criterion. Besides output validation, other quality criteria for agent-based models include for instance input validation or process validation, reflecting the realism of the initialization and the mechanisms implemented in the model.
Qualitative validation procedures
The brief introduction is by no means an exhaustive summary of the broad discussion on validation. Already the measurement of empirical data can be put into question. Less discussed however, had been the role which qualitative methods potentially could play in this endeavor. In fact, there has been a long debate in the community of qualitative social research on this issue as well. Like agent-based social simulation also qualitative methods are challenged by the notion of validation. It has been noted that already the vocabulary that is used in attempts to ensure scientific rigor has a background in a positivist understanding of science whereas qualitative researcher often take up constructivist or poststructuralist positions (Cho and Trent 2006). For this reason, in qualitative research sometimes the notion of trustworthiness (Lincoln and Guba 1985) is preferred rather than speaking of validation. In an influential article (according to google scholar cited more than 17.000 times in May 2023) Creswell and Miller (2000) distinguish between a postpositivist, a constructivist, and a critical paradigm as well as between the lens of the researcher, the lens of the study participants, and the lens of external people and assign different validity procedures for qualitative research to the combinations of these different paradigms and lenses.
Paradigm/ lens
postpositivist
constructivist
critical
Lens of researcher
triangulation
Disconfirming evidence
Reflexivity
Lens of study participants
Member checking
Engagement in the field
Collaboration
Lens of external people
Audit trial
Thick description
Peer debriefing
Table 1. validity procedures according to Creswell and Miller (2000).
While it remains contested if the validation procedure depends on the research design, this is at least a source of different accounts. Others differentiate between transactional and transformational validity (Cho and Trent 2006). The former concentrates on formal techniques in the research process for avoiding misunderstandings. Such procedures include for instance, techniques such as member checking. The latter account perceives research as an emancipatory process on behalf of the research subjects. This goes along with questioning the notion of absolute truth in the domain of human sciences which calls for alternative sources for the legitimacy of science such as emancipation of the researched subjects. This concept of emancipatory research resonates with participatory modelling approaches. In fact, in participatory modelling accounts some of these procedures are well-known even though they differ in terminology. The participatory approach originates from research on resource management (Pahl-Wostl 2002). For this purpose, integrated assessment models have been developed, inspired by the concept of post-normal science (Funtowicz and Ravetz 1993). Post-normal science emphasizes the communication of uncertainty, justification of practice, and complexity. This approach recognizes the legitimacy of multiple perspectives on an issue, both with respect to multiple scientific disciplines as well as lay men involved in the issue. For instance, Wynne (1992) analyzed the knowledge claims of sheep farmers in the interaction with scientists and authorities. In such an extended peer community of a citizen science (Stilgoe 2009), lay men of the affected communities play an active role in knowledge production, not only because of moral principles of fairness but to increase the quality of science (Fjelland 2016). One of the most well-known participatory approaches is the so-called companion modelling (ComMod) developed at CIRAD, a French agricultural research center for international development. The term companion modelling has been coined originally by (Barreteau et al 2003) and been further developed to a research paradigm for decision making in complex situations to support sustainable development (Étienne 2014). In fact, these approaches have a strong emancipatory component and rely on collaboration and member checking for ensuring resonance and practicality of the models (Tesfatsion 2021).
An interpretive validation procedure
While the participatory approaches show a convergence of methods between modelling and qualitative research even though they differ in terminology, in the following a further approach for examining the trustworthiness of simulation scenarios will be introduced that has not been considered so far, namely interpretive methods from qualitative research. A strong feature of agent-based modelling is that it allows for studying “what-if” questions. The ex-ante investigation of possible alternative futures enables identifying possible options of action alternatives but also detecting early warning signals of undesired developments. For this purpose, counterfactual scenarios are an important feature of agent-based modelling. It is important to note in this context that counterfactuals do not match empirical data. In the following it is suggested to examine the trustworthiness of counterfactual scenarios by using methods from objective hermeneutics (Oevermann 2002), the so-called sequence analysis (Kurt and Herbrik 2014). In terms of Creswell and Miller (2000) the examination of trustworthiness is from the lens of the researcher and a constructivist paradigm. For this purpose, simulation results have to be transformed into narrative scenarios, a method which is described in (Lotzmann and Neumann 2017).
In the social sciences, sequence analysis is regarded as the central instrument of hermeneutic interpretation of meaning. It is “a method of interpretation that attempts to reconstruct the meaning of any kind of human action sequence by sequence, i.e. sense unit by sense unit […]. Sequence analysis is guided by the assumption that in the succession of actions […] contexts of meaning are realized …” (Kurt and Herbrik 2014: 281). A first important rule is the sequential procedure. The interpretation takes place in the sequence that the protocol to be analyzed itself specifies. It is assumed that each sequence point closes possibilities on the one hand and opens new possibilities on the other hand. This is done practically by sketching a series of stories in which the respective sequence passage would make sense. The basic question that can be asked of each sequence passage can be summarized as, “Consider who might have addressed this utterance to whom, under what conditions, with what justification, and what purpose?” (Schneider 1995: 140). The answers to these questions are the thought-experimentally designed stories. These stories are examined for commonalities and differences and condensed into readings. Through the generation of readings, certain possibilities of connection to the interpreted sequence passage become visible at the same time. In this sense, each step of interpretation makes sequentially spaces of possibility visible and at the same time closes other spaces of possibility.
In the following it will be argued that this method enables an examination of the trustworthiness of counterfactual scenarios using the example of a counterfactual simulation scenario of a successful non-violent conflict regulation within a criminal group: ‘They had a meeting at their lawyer’s office to assess the value of his investment, and Achim complied with the request. Thus, trust was restored, and the group continued their criminal activities’ (names are fictitious). Following Dickel and Neumann (2021) it is argued that this is a meaningful story. It is an example of how the linking of the algorithmic rules generates something new from the individual parts of the empirical material. However, it also shows how the individual pieces of the puzzle of the empirical data material are put together to form a collage that tells a story that makes sense. A sequence that can be interpreted in a meaningful way is produced. It should be noted, however, that this is a counterfactual sequence. In fact, a significantly different sequence is found in the empirical data: ‘Achim was ordered to his lawyer’s office. Instead of his lawyer, however, Toby and three thugs were waiting for him. They forced him to his knees and pointed a machine gun at his stomach’. In fact, this was by no means a non-violent form of conflict regulation. However, after Achim (in the real case) was forced to his knees by three thugs and threatened with a machine gun, the way to non-violent conflict regulation was hardly open any more. The sequence generated by the simulation, on the other hand, shows a way how the violence could have been avoided – a way that was not taken in reality. Is this now a programming error in the modeling? On the contrary, it is argued that it demonstrates the trustworthiness of the counterfactual scenario: from a methodological point of view a comparison of the factual with the counterfactual is instructive: Factually, Achim had a machine gun pointed at his stomach. Counterfactually, Achim agreed on a settlement. From a sequence-analytic perspective, this is a logical conclusion to a story, even if it does not apply to the factual course of events. Thus, the sequence analysis shows that the simulation here has decided between two possibilities, a path branching in which certain possibilities open and others close.
The trustworthiness of a counterfactual narrative is shown by whether 1) a meaningful case structure can be generated at all, or whether the narrative reveals itself as an absurd series of sequence passages from which no rules of action can be reconstructed. 2) it can be tested whether the case structure withstands a confrontation with the ‘external context’ and can be interpreted as a plausible structural variation. If both are given, scenarios can be read as explorations of the space of cultural possibilities, or of a cultural horizon (in this case: a specific criminal milieu). Thereby the interpretation of the counterfactual scenario provides a means for assessing the trustworthiness of the simulation.
References
Barreteau, O., et al. (2003). Our companion modelling approach. Journal of Artificial Societies and Social Simulation 6(2): 1. https://www.jasss.org/6/2/1.html
Dickel, S., Neumann. M. (2021). Hermeneutik sozialer Simulationen: Zur Interpretation digital erzeugter Narrative. Sozialer Sinn 22(2): 252-287. https://doi.org/10.1515/sosi-2021-0013
Fjelland, R. (2016). When Laypeople are Right and Experts are Wrong: Lessons from Love Canal. International Journal for Philosophy of Chemistry 22(1): 105–125. https://www.hyle.org/journal/issues/22-1/fjelland.pdf
Kurt, R.; Herbrik, R. (2014). Sozialwissenschaftliche Hermeneutik und hermeneutische Wissenssoziologie. In: Baur, N.; Blasius, J. (eds.): Handbuch Methoden der empirischen Sozialforschung, pp. 473–491. Springer VS, Wiesbaden. https://link.springer.com/chapter/10.1007/978-3-658-21308-4_37
Lotzmann, U., Neumann, M. (2017). Simulation for interpretation. A methodology for growing virtual cultures. Journal of Artificial Societies and Social Simulation 20(3): 13. https://www.jasss.org/20/3/13.html
Lincoln, Y.S., Guba, E.G. (1985). Naturalistic Inquiry. Sage, Beverly Hill.
Pohl-Wostl, C. (2002). Participative and Stakeholder-Based Policy Design, Evaluation and Modeling Processes. Integrated Assessment 3(1): 3-14. https://doi.org/10.1076/iaij.3.1.3.7409
Schneider, W. L. (1995). Objektive Hermeneutik als Forschungsmethode der Systemtheorie. Soziale Systeme 1(1): 135–158.
Stilgoe, J. (2009). Citizen Scientists: Reconnecting Science with Civil Society. Demos, London.
Tesfatsion, L. (2021). “Agent-Based Modeling: The Right Mathematics for Social Science?,” Keynote address, 16th Annual Social Simulation Conference (virtual), sponsored by the European Social Simulation Association (ESSA), September 20-24, 2021.
Wynne, B. (1992). Misunderstood misunderstanding: social identities and public uptake of science. Public Understanding of Science 1(3): 281–304.
Neumann, M. (2023) The Challenge of Validation. Review of Artificial Societies and Social Simulation, 18th Apr 2023. https://rofasss.org/2023/04/18/ChallengeValidation
1Escuela de Ciencias Empresariales, Universidad Católica del Norte, Coquimbo, Chile, and CESIMO, Universidad de Los Andes, Mérid.
2CEMISID, Universidad de Los Andes, Merida, Venezuela; GIDITIC, Universidad EAFIT, Medellin, Colombia; and Universidad de Alcala, Dpto. Automatica, Alcala de Henares, Spain.
Abstract.
This work suggests to complementarily use Multi-Fuzzy Cognitive Maps (MFCM) and Multi-agent Based Simulation (MABS) for social simulation studies, to overcome deficiencies of MABS for contextually understanding social systems, including difficulties for considering the historical and political domains of the systems, variation of social constructs such as goals and interest, as well as modeler’s perspective and assumptions. MFCM are a construction much closer than MABS to natural language and narratives, used to model systems appropriately conceptualized, with support of data and/or experts in the modeled domains. Diverse domains of interest can be included in a MFCM, permitting to incorporate the history and context of the system, explicitly represent and vary agents’ social constructs, as well as take into account modeling assumptions and perspectives. We briefly describe possible forms of complementarily use these modeling paradigms, and exemplifies the importance of the approach by considering its relevance to investigate othering and polarization.
1. Introduction
In order to understand better issues such as othering and polarization, there is a claim in social simulation for research that includes the important domains of history, politics and game of power, as well as for greater use of social science data, make more explicit and conscious about the models’ assumptions, and be more cautious in relation to the interpretation of the simulations’ results (Edmonds et al., 2020). We describe a possible form of dealing with these difficulties: combining Multi-Agent based Simulation (MABS) and Multi Fuzzy Cognitive Maps (MFCM) (or other forms of cognitive maps), suggesting new forms of dealing with complexity of social behavior. By using MFCM an alternative modeling perspective to MABS is introduced, which facilitates expressing the context of the model, and the modelers’ assumptions, as suggested in Terán (2004). We will consider as a case studying othering and polarization, given the difficulties for modeling it via MABS (Edmonds et al., 2020). Our proposal permits to explicitly represent social constructs such as goals, interest and influence of powerful actors on, e.g., people’s othering and polarization, and so in better contextualizing the simulated model. Variations of social constructs (e.g., goals, othering, polarization, interests) can be characterized and modeled by using MFCM.
Combined use of MABS and Fuzzy Cognitive Maps (FCM) (MFCM, multi FCM, are an extension of FCM, see the Annex) has already been suggested, see for example Giabbanelli (2017). MABS develop models at the micro level, while FCM and MFCM permits us to create models at the macro or contextual level; the idea is to use one to complement the other, i.e., to generate rich feedback between them and enhance the modeling process. Additionally, Giabbanelli propose FCM as a representation closer than MABS to natural language, allowing more participatory models, and better representation of the decision making process. Giabbanelli recommend forms of combining these two modeling approaches, highlighting key questions modelers must be careful about. In this line, we also propose a combined usage of a MFCM and MABS to overcome deficiencies of MABS modelling in Social Simulation.
Initially (in section 2) we offer a description of human societies from a broad view point, which recognizes their deep complexities and clarifies the need for better contextualizing simulation models, allowing modeling of diverse agents’ constructs, and making explicit modelers’ assumptions and perspectives. Afterwards (in section 3), we briefly review the drawbacks of MABS for modeling some of these deep complexities. Then (in section 4), MFCM are briefly described, supported on a brief technical account in the Annex. Following (in section 5), we suggest to complementarily use MABS and MFCM for having a more comprehensive representation of human societies and their context, e.g., to better model problems such as othering and polarization. MFCM will model context and give a conceptual mark for MABS (allowing to model variation of context, e.g., changes of agents’ interests or goals, making explicit modelers’ perspective and assumptions, among other advantages), which, in turn, can be used to explore in detail specific configurations or scenarios of interest suggested by the MFCM. Finally (in section 6), some conclusions are given.
2. (A wide view of) Human societies and influence of communication media on actual culture
As humans and primates, we recognise the social groups within which we develop as people (e.g., family, the community where we grow up, partners at the school or at work) as part of our “large home”, in which its members develop a common identity, with strong rational and emotional links. Other groups beyond these close ones are “naturally” estrangers for us and its members “instinctively” seen as others. In large civilizations such as western society, we extend somewhat these limits to include nations, in certain respects. In groups we develop perspectives, follow certain myths and rites, and have common interests, viewpoints about problems, solutions for these, and give meaning to our life. Traditionally, human societies evolve from within groups by direct face to face interaction of its members, with diverse perspectives, goals, interest, and any other social construct with respect to other groups. Nowadays this evolution mainly from natural interaction has been importantly altered in some societies, especially western and western influenced societies, where social media has introduced a new form of communication and grouping: virtual grouping. Virtual grouping consists in the creation of groups, either formally or informally, by using the internet, and social networks such as Facebook, Instagram, etc. In this process, we access certain media sites, while discarding others, in accordance with our preferences, which in turn depends on our way of thinking and preferences created in social, both virtual and direct (face to face), interaction. Currently, social media, and traditional media (TV, newspapers, etc.) have a strong influence on our culture, impacting on ours myths, rites, perspectives, forms of life, goals, interests, opinion, reasoning, emotions, and othering.
Characteristics of virtual communication, e.g., anonymity, significantly impacts on all these constructs, which, differently from natural interaction, have a strong potential to promote polarization, because of several reasons, e.g., given that virtual environments usually create less reflexive groups, and emotional communication is poorer or lack deepness. Virtual interaction is poorer than direct social interaction: the lack of physical contact strongly reduces our emotional and reflexive connection. Virtual social interaction is “colder” than direct social interaction; e.g., lack of visual contact stops communication of many emotions that are transmitted via gestures, and prevents the call for attention from the other that visual contact and gestures demands.
Even more, many times sources and veracity of information, comments, ideas, and whatever is in social media, are not clear. Even more, fake news are common in social media, what generate false beliefs, and behavior of people influenced and somewhat controlled by those who promote fake news. Fake news can in this sense generate polarisation, as some groups in the society prefer certain media, and other groups choose a different one. As these media may promote different perspectives following interest of powerful actors (e.g., political parties), conflicting perspectives are induced in the different groups, what in turn generates polarization. Social media are highly sensitive to manipulation by powerful actors worldwide, including governments (because of, e.g., their geopolitical interests and strategies), corporations (in accordance with their economic goals), religious groups, political parties, among many others. Different groups of interests influence in direct and indirect, visible and hidden, forms the media, following a wide diversity of strategies, e.g., those of business marketing, which are supported by knowledge of people (e.g., psychology, sociology, games theory, etc.). Thus, the media can create and contribute to create visions of the word, or perspectives, in accordance with the interest of powerful international or national actors. For more about all this, see, e.g, Terán and Aguilar (2018).
As a consequence, people following media that promotes a world view, related with some powerful actor(s) (e.g., a political party or a group of governments) virtually group around media that support this world view, while other people do the same in relation to other media and powerful actor(s), who promote(s) a different perspective, which many times is in conflict with the first one. Thus, grouping following the media sometimes promotes groups with conflicting perspectives, goals, interests, etc., which generates polarization. We can find examples of this in diverse regions and countries of the world. The media has important responsibility for polarization in a diversity of issues such as regional integration in Europe, war in Ukraine, migrations from Middle East or Africa to Europe, etc. Consequently, media manipulation sometimes allow powerful actors to influence and somewhat control perspectives and social behavior. Even more, the influence of social media on people is sometimes stronger than the influence of direct social interaction. All these introduce deep complex issues in social human interaction and behavior. This is why we have chosen polarization as the case study for his essay.
Consequently, to comprehend actual human behavior, and in particular polarization, it is necessary to appropriately take into account the social context, what permits to understand better the actual complexity of social interaction, e.g., how powerful international, national, and local actors’ influence on media affects people perspectives, goals, interest, and polarization, as well as their strategies and actions in doing so. Contextualized modeling will help in determining social constructs (goals, interests, etc.) in certain situations, and their variation from situation to situation. For this, we suggest complementing MABS with MFCM. For more about the consequences of virtual interaction, see for example Prensky (2001a, 2001b). Prensky (2009) has also suggested forms to overcome such consequences: to promote digital wisdom. MABS and MFCM models will help in defining forms of dealing with the problems of high exposure to social networks, in line with Prensky’s concerns.
3. Weakness of the MABS approach for modeling context
Edmonds et al. (2020) recognize that MABS models assume a “state of the world” or “state of nature” that does not include the historical context of the agent, e.g., in such a way that they explicitly present goals, interests, etc., and pursue them via political actions, sometimes exerting power over others. For instance, the agents can not change their goals, interests or desires during the simulation, to show certain evolution, as a consequence of reflection and experience at the level of desires, allowing cognitive variations. The models are strongly limited in relation to representing the context of the social interaction, which in part determines variation of important factors of agents’ behavior, e.g., goals. This, to a good extent, is due to lack of representation of the agents’ context. For the same reason, it is difficult to represent modelers assumptions and perspectives, which might also be influenced by social media and powerful actors, as explained above.
The Special Issue of the Social Science Computer Review. (Volume 38, Issue 4, August 2020, see for instance Edmonds et al. (2020) and Rocco and Wander (2020)), presents several models aiming at dealing with some of these drawbacks of MABS, specifically, to relate models to social science data, be more aware about the models’ assumptions, and be more cautious in relation to the interpretation of the simulations’ results. However, in these works diverse difficulties are not addressed, e.g., having appropriate representation of the context in order to explicitly consider diverse constructs, e.g., goals and interests, as well as having a wide representation of modelers perspectives and assumptions so that diverse perspectives can be addressed and compared, among other important matters.
MABS represent social interaction, i.e., the interaction in a group, where the agent’s goal, and other social constructs are assumed given, not variable, and to understand the context where they appear is not of interest or is out of reach (too difficult). However, as explained above, agents are in diverse social groups, not only in the simulated one, and so goals, interests, and beliefs in the modeled group are shaped in accordance to their interactions in diverse groups, and the influence of multiple, virtual and natural groups in which they participate. In order to represent variations of such elements, the context must be taken into account, as well as to elaborate models from narratives. MFCM is naturally close to narratives, as it is elaborated from conceptual frameworks. In this sense, MFCM might represent an intermediate step towards MABS models. In a MFCM and in the steps towards elaborating the MABS, modelers’ perspectives and assumptions can be made explicit. In addition, MABS presents limitations to determine the conditions for which a certain behavior or tendency occurs (Terán, 2001; Terán et al. 2001), i.e., for making strong inferences and theorem proving of tendencies for subsets of the theory of the simulation, which could more easily be performed in the MFCM. Hopefully, exploring configurations of the MFCM the proof could be carried out indirectly, in a higher level than in MABS, as has already been suggested in previous papers (Aguilar et al., 2020; Perozo et al., 2013).
4. Multi-Fuzzy Cognitive Maps (MFCM)
We suggest conceptual or cognitive maps as a more flexible form than MABS to represent context of a social situation, and in particular MFCM, as implemented by Aguilar and others (see e.g., Kosko, 1986; Aguilar 2005, 2013, 2016; Aguilar et al., 2016, 2020; Contreras and Aguilar, 2010; and Sánchez et al., 2019; Puerto et al., 2019). A brief description of Fuzzy and Multi-fuzzy cognitive maps, following Sánchez et al. (2019), is given in the Annex.
Fuzzy cognitive maps help us in describing the context via qualitative (e.g., very low, low, medium, high, too high) and quantitative variables, as indicated in the annex. The system is represented by the network of concepts (variables) interrelated via weights (also given by variables). The high level of the MFCM paradigm, differently from a MABS, permits us to explicit different elements of the models such as the agents constructs (goals, interests, etc.), as well as the modelers assumptions and perspectives, as suggested in Terán (2004). MFCM will facilitate to explicit the accumulated set of assumptions (“abstraction-assumptions, design-assumptions, inference-assumptions, analysis-assumptions, interpretation-assumptions and application-assumptions”, as these are summarized in Terán, 2004).
In a MFCM, a particular situation of the system is given by a specific configuration of the weights (see, e.g., Sánchez et al., 2019). Suppose we are dealing with a model similar to that elaborated in Sánchez et al. (2019) to study the quality of opinion in a community. Sánchez et al. examine the capabilities of the MFCM for knowledge description and extraction about opinions presented in a certain topic, allowing the assessment of the quality of public opinion. Special attention is offered to the influence of the media on public opinion. The evolution of the concepts and relationships is presented. Concepts define the relevant aspects from which public opinion emerges, covering diverse domains, for instance, the social, technological and psycho-biological ones. The MFCM permits to identify the media preferred by the public in order to better understand several issues, including the high esteem that the new communication media hold.
In line with this, let us assume that we want to understand the quality of public opinion in a community of Europe about diverse issues during 2022. This network, with a certain configuration of the weights, but unspecified concepts, represents a social system with a certain structure (as the weights are given) that is in some sense general, as the values of the concepts can still vary. Variation of the concepts represents different scenarios of the social system (with a given structure, defined by the weights); e.g., the model of a European community considered in relation to three scenarios regarding the state of public opinion in relation to specific issues: 1: climate change, 2: situation of tourism in the community, 3: secondary effects of the COVID-19 vaccines. The weights of the network are determined by using a variety of scenarios; i.e., the network is trained with several scenarios, for which all possible values of the concepts are known. Once the network is trained, it can be used to infer unknown specific values of the concepts for other scenarios (following Aguilar et al. 2020; Sánchez et al. 2019; Terán y Aguilar, 2018); e.g., the state of public opinion in relation to the involvement of EU in the war in Ukraine. Even more, by exploring an appropriate set of scenarios, proofs about the state of certain concepts can be developed; e.g., that a majority of people in the community is against direct EU involvement in the war in Ukraine. The proof could be carried out for a subset of the possible configurations of a domain, several domains, or part of a domain, e.g., for the psycho-biological domain. Additionally, having an appropriate elaboration of the model would allow evaluating how polarized is the opinion of the community in relation to the involvement of EU in that war.
Diverse configurations of the MFCM can represent different modelers’ perspectives and assumptions, as well as various agents’ constructs, such as goals, interests, etc., allowing to deal with the above described drawbacks of MABS to cope with complexity of social systems.
5. Combined use of MABS and cognitive maps
The combined usage will give at least two levels of modeling: the inner, defined by the agents’ interaction concreted in a MABS, and the outer or contextual one, given by the MFCM. These will be the two last levels in the description given in 5.1. Interaction between these models occurs as the modeler interprets each model outputs and feedbacks the other. Ideally, we would have direct automatic feedback between these models.
5.1 Levels of description of the System
In order to contextually model a social system and investigate problems such as polarization, we suggest below five levels of description of the system. The first three levels are not directly associated to computational models, while levels four and five are descriptions that assist development of the computational models: MFCM and MABS, respectively. Each level gives context to the following one (the first gives context to the second, etc.). A lower level (e.g., 1. in relation to 2.) of description corresponds to a more general language, as suggested in Terán (2004). Each level must take into account the previous levels, especially the immediately superior level, which gives the most immediate context to it. This description is in line with the suggestions given in Terán (Idem). Each description makes certain assumptions and is shaped by the modeler’s perspective, which in part is coming from those actors given information to build the model. Assumptions and perspectives introduced in level of modeling i, i = 1, …, 6, can be called Assumptions-given in (i) and Perspectives-given in (i). At levels of description j, Assumption i = 1, 2, …, i are accumulated, and can be called Assumptions(j), as well as holistic Perspective(j) based on Perspectives-given in (i), i = 1, 2, .., i. These assumptions and perspectives correspond to those defined in Terán (2004) as “abstraction-assumptions, design-assumptions, inference-assumptions, analysis-assumptions, interpretation-assumptions and application-assumptions”.
Describe in natural language the system, including its relevant history, with emphasis in culture (practices, costumes, etc.) and behavior of individuals and groups relevant to the object of the study, e.g., from a historical-ontological perspective. Here, how the system has reached its actual situation is explained. This will give a global view of the society and the general form of behavior, problematics, conflicts, etc.
Describe the diverse relevant domains given context to the system of interest in accordance with the study, e.g., political, economical, dominant actors, etc., and the relationships among them. Concrete specifications of these domains sets scenarios for the real system, i.e., possible configurations of it.
Describe the particular social group of interest as part of the society explained above, in 1., and the domains given in 2., showing its particularities, e.g., culturally, in terms of interests, situation and social interaction of this group in relation with other groups in the whole society, in accordance to the problematic addressed in the study.
Elaborate a cognitive map of the situation of the social group of interest, following the description given in 2 and 3. This is a description to be represented in a computational language, such as the MFCM tool developed by Contreras and Aguilar (2010).
Describe the MABS model. The MABS model is then represented in a simulation language.
The computational MFCM (or other cognitive map) and the MABS developed following 4. and 5. are then used to generate the virtual outputs and simulation study.
5.2 Possible combined uses of MFCM and MABS.
The MFCM (in general a cognitive or conceptual map) gives context to the MABS model (modeler’s assumptions and perspectives are added in the process, as indicated above), while the MABS model represents in detail the interaction of the agents’ considered in the MFCM for a specific scenario of this, as indicated in the levels of modeling given above. With this idea in mind, among the specific forms a combined usage of ABM and MFCM, we have:
i) Offering feedback from the MFCM to the MABS. A MABS in a certain configuration can be used to generate either directly or indirectly (e.g., with additional verification or manipulations) input for a simulation model. For example, in parallel to the model presented in Sánchez et al. (2019), where the domains social, biological-psychological, technological and state of the opinion are displayed, a MABS model can be developed to represent the interaction between social entities, such as people who receive information from the media, the media itself, and powerful actors who design the agenda setting of the media. This MABS model might use diverse methodologies, e.g., endorsements or BDI, to represent social interaction, or a higher level of interaction where actors share resources, on which they have interest, as in a System of Organized Action (see, e.g., SocLab models Terán and Sibertin-Blanc, 2020; Sibertin-Blanc et al., 2013). Constructs required as inputs for the model, e.g., goals, interests, values to define the endorsements schema, etc. could be deducted from the MFCM, as direct values of some concepts or functions (mathematical, logical, etc.) of the concepts. These operations could be defined by experts in the modeled domains (e.g., media owners, academics working in the area, etc.). Ideally, we would have an isomorphic relation between some variables of the MABS and variables of the MFCM – however, this is not the usual case –. In this process, the MABS is contextualized by the MFCM, whose modeling level also permits to identify modeler’s assumptions and perspectives. Also, in a narrative, and then in the MFCM, goal, interest, and other constructs can be explicitly represented, then varied and their consequence understood in order to feedback the MABS model.
ii) Giving feedback to the MFCM from the MABS. Inputs and outputs values of the MABS simulation can be used as an input to the MFCM, e.g., as a set of scenarios to train the network and determine a certain structure of the MFCM, or to determine a specific scenario/ configuration (where both, the weights and the concepts are known).
iii) Determining conditions of correspondence among the models. By simulating the MABS associated to certain scenarios of the MFCM, or, vice-verse, by determining the scenarios of MFCM related to certain MABS, the consistency among the two models and possible errors, omissions, etc. in one of the models can be detected, and then the corrections applied. Even more, this exercise can hopefully determine certain rules or conditions of correspondence among the MABS and the MFCM.
iv) Using a model to verify properties in the other model. Once certain correspondence among the models has been determined, we can use one of the models to help in determining properties of the other. For instance, a proof of a tendency in an MABS (this has been an important area of research, Terán, 2001; Terán et al. 2001; Edmonds et al., 2006) could be developed in a much easier form in the corresponding MFCM. For this, we need to characterize the set of configurations of the MFCM corresponding to the set of configurations of the MABS for which we want to perform the proof.
These possible combined uses of MFCM and MABS do not exhaust all potentials, and diverse other alternatives could appear in accordance with the needs in a particular study. Even more, automatic feedback between MFCM (or other cognitive or conceptual map) and MABS could be implemented in the future, to facilitate the mutual contributions between the two modeling approaches. This would cover modeling requirements the MABS in itself does not support at present.
5.3 A case: Modeling othering and polarization,the case of “children with virtually mediated culture”.
We outline a possible model that considers othering and polarization. In section 2 we described a society. In a society, as virtual groups become homogeneous in beliefs, motivations, intentions and behavior, certain sort of endogamy of ideas and opinions appear, constraining the variety and richness of perspectives from which people observe and judge others, making them generally less tolerant to others, more restrictive in accepting opinions and behavior of others, so less inclusive. This has diverse additional effects, for instance, increase of polarization between virtual groups regarding a diversity of themes. Problems such as polarization occur also in children with strong usage of virtual social networks (see, Prensky, 2001a, 2001b, 2009).
To investigate this problem and support the MABS, as a case, we suggest a MFCM with four levels (see Figure 1). The goal of the models (MFCM and MABS) would be to better understand the differences between the communities of children whose interaction is basically virtually mediated and the community of children whose interaction is face to face, or direct, people to people. In general, it is of interest to determine the state of othering and polarization for diverse configurations of the MFCM. As we explained above, there are clear differences between virtual and face to face interaction, consequently the upper layers in Figure 1 are the two possible niches of cultural acquisition (costumes, points of view, etc.), during life of people (ontogenesis), namely, the virtual mediated culture and the direct, face to face, cultural acquisition. These two layers involve interaction among diverse actors (e.g., people, media and powerful actors are present in layer 2). Layer 2 represents technological actors, while layer 1 represents social interaction, but both of them might involve other elements, if required. The third level represent those biological aspects related with behavior, which are created via culture: the psycho-biological level. Both levels, 1 and 2, affect the third layer, as emotions, reasoning, etc., are founded on people interaction and have a cultural base. Constructs of behavior such as goals, interests, desires, polarization, etc., appear and can be explicitly represented at this level. This third level, in turn, impacts on the overall state of the community, e.g., on the auto-generative capacity of the society, finally affecting global society/community’s othering and polarization, as our emotions, reasoning, etc., impact on our view point, on othering, etc. These last are variables defined in terms of the previous levels. In this model, the definition of concepts such as “othering” and “polarization” is crucial, and indicates basic modeling assumptions and perspective. Finally, the overall state of society/community impacts back on the cultural niches (layers 1 and 2).
In a specific situation, the whole interaction (1) of the society or community is divided between the two niches given by layers 1 and 2, a proportion of interaction frequency occurs as virtual communication, and the compliment (one minus the proportion of virtual interaction), occurs as direct, face to face, contact. This is represented in Figure 3 by the variable “Proportion of virtual interaction type”. Changes of this variable allows us to explore diverse configurations or scenarios of interaction, ranging from total virtual interaction (null direct contact) (the variable takes the value 1), to null virtual interaction (total direct contact) (the variable takes the value 0).
Figure 1. The four levels of the MFCM. The two alternative niches structuring the psycho-biology of people are at the top of the process. The overall state represents general measures such as the auto-generative character of a social system, and attitudes including othering and polarization.
Example of possible variables in some levels (Figure 1) are:
i) Face to face interaction, and ii) virtual interaction or technological: The next variables are candidates to be at these levels. Degree of:
Coherence of the interaction (possible state: good, etc.);
Identification of the others in the interaction (good or clear, etc.);
Richness of the interaction (high or good, etc.);
Truthfulness of the messages (fairness) (e.g., good: messages and communication are fair);
Openness of the community (e.g., high: usually people is open to interact with others);
Speed of the interaction (e.g.: low, medium, …);
Intentioned influence and control of the communication by powerful actors (e.g., high, medium, low, ..);
iii) Psycho-biological level:
Reflection (state: good means that people question their experiences, and observed phenomena);
Closeness of interpretations, attitudes, desires, intentions, and plans (a high value means that people’s interpretations, etc., are not very different);
Emotion and mood;
Empathy;
Addiction to virtual interaction;
Goal;
Interest;
Immediatism (propensity to do things quickly and constantly change focus of reasoning).
iv) Overall state of people and society:
Auto-generative capacity of the society;
Capacity of people to reflect about social situations (and autonomously look for solutions);
Othering;
Polarization;
Concepts at each layer impacts concepts at the other layers. E.g., concepts of level three have a strong impact on concepts of the fourth layer, such as “polarization”, and “auto-generative character of the society”.
As indicated above, to understand the dynamics of the MFCM we can develop a wide range of scenarios, for instance, varying the switch “Proportion of virtual interaction” in the interval [0, 1], to explore a set of scenarios for which the degree of virtual interaction increases from 1 to 0, as the proportion of direct interaction decreases from 1 to 0 (the real case corresponds to an intermediate value between 1 and 0). These experiments will help us in understanding better the consequences of virtual mediated culture. Even when the outline of the model presented here might need some adjustments and improvements, the present proposal keeps its potential to reach this goal.
The MFCM will be useful to deal with many issues and questions of interest, for instance:
How social networks affect basic social attitudes such as: i) critical rationality (people’s habit for questioning and explaining their experience (issues/phenomena in their life)), ii) tolerance, iii) compromise with public well-being, and iv) othering and polarization
How social networks affect social feelings, such as empathy.
The MABS model will be elaborated in accordance with the description of the MFCM indicated above. In particular, different values of the social constructs at levels 1 and 2 (e.g., goals and interests of the actors), and the corresponding state of layer 3 (e.g., Polarization), imply diverse MABS models.
The whole network of concepts, the particular network of concepts at each level, and the definition of each concept, offers a perspective of the modelers. Different modelers can develop these elements of the model differently. Assumptions can be identified also at each level. Both, perspectives and assumptions come from the modelers as well as from the theories, consult to experts to create the model, etc.. An specific model is not part of this essay, but rather a subject of future work.
6. Conclusion
Social simulation has been widely recognized as an alternative to study social systems, using diverse modeling tools, including MABS, which, however, present some limitations, like any other research tool. One of the deficiencies of MABS is their limitations to contextually modeling social systems; e.g., to suitably include the historical and political contexts or domains; difficulties to represent variation of agents’ constructs, e.g., goals and interest; and drawbacks to made explicit modeler’s assumptions and perspectives. In this paper, we have suggested to mutually complement MABS and MFCM, to overcome MABS drawbacks, to potentiate the usefulness of MABS to represent social systems.
We argue that the high level of the MFCM paradigm permits us to express different elements of the models such as the agents’ constructs (goals, interests, etc.), as well as the modelers assumptions and perspectives, as suggested in Terán (2004). Thus, MFCM facilitates the identification of the accumulated set of assumptions during the modeling process. Even more, diverse configurations of a MFCM can represent diverse modelers’ perspectives and assumptions, as well as diverse agents constructs, such as goals, interests, etc., allowing to deal with the above described complexity. This permits us to more realistically elaborate models of a wide diversity of social problems, e.g., polarization, and consequences of the influence of social networks in culture.
Among the forms MFCM and MABS complement each other we have identified the followings: mutual feedbacking of variables and concepts between the MFCM and the MABS, determining conditions of correspondence among the models, what facilitate other modeling needs, e.g., using a model to verify properties in the other model (e.g., proofs required in a MABS could be carried out in a corresponding MFCM).
A case study was outlined to exemplify the problematic that can be addressed and the advantages of using MFCM to complement MABS: Modeling othering and polarization, the case of children with virtually mediated culture. Combined use of MFCM and MABS in this case will contribute to understand better the problems created by the high use of digital interaction, especially social networks, as described by Prensky (2001a, 2001b, 2009), given that virtual interaction has strong influence on our culture, impacting on ours myths, rites, perspectives, goals, interests, opinion, othering and polarization, etc. Characteristics of virtual communication, e.g., anonymity, significantly impacts on all these constructs, which, differently from natural interaction, have a strong potential to promote certain tendencies of such constructs, e.g., polarization or our opinions, because of several reasons, for instance, given that virtual environments usually create less reflexive groups, while emotional communication is poorer or lack deepness. It is difficult to represent all these dynamics in a MABS, but it can be alternatively expressed in a MFCM.
The purpose of the work was to give an outline of the proposal; future work will be conducted to wholly develop concrete study cases with complementary MABS and MFCM models.
References
Aguilar, J. (2005). A Survey about Fuzzy Cognitive Maps Papers. International journal of computational cognition 3 (2), pp. 27-33.
Aguilar, J. (2013) Different Dynamic Causal Relationship Approaches for Cognitive Maps”, Applied Soft Computing, Elsevier, 13(1), pp. 271–282.
Aguilar, J. (2016) “Multilayer Cognitive Maps in the Resolution of Problems using the FCM Designer Tool”, Applied Artificial Intelligence, 30 (7), pp. 720-743.
Aguilar J., Hidalgo J., Osuna F., Perez N.(2016) Multilayer Cognitive Maps to Model Problems”, Proc. IEEE World Congress on Computational Intelligence, pp. 1547-1553, 2016.
Aguilar Jose, Yolmer Romero y OswaldoTerán (2020). “Analysis of the effect on the marketing of the sport product from the “par Conditio” principle in Latin American football and baseball competitions”. Submitted to International Journal of Knowledge-Based and Intelligent Engineering Systems.
Contreras, J. Aguilar, J. (2010). “The FCM Designer Tool”. Fuzzy Cognitive Maps: Advances in Theory, Methodologies, Tools and Application (Ed. M. Glykas), Springer, pp. 71-88, 2010.
Edmonds, B. and Hales, D. and Lessard-Phillips, L.(2020). Simulation Models of Ethnocentrism and Diversity: An Introduction to the Special Issue. Social Science Computer Review. Volume 38 Issue 4, August 2020, 359–364, https://journals.sagepub.com/toc/ssce/38/4
Edmonds Bruce, Oswaldo Terán y Grary Polhill (2006). “To the Outer Limits and Beyond –characterising the envelope of social simulation trajectories”, Proceedings of theThe First World Congress on Social Simulation WCSS”, Kyoto, 21-25 Agosto, 2006.
Giabbanelli Philippe, Steven Gray and Payam Aminpour (2017), Combining fuzzy cognitive maps with agent-based modeling: Frameworks and pitfalls of a powerful hybrid modeling approach to understand human-environment interactions, Environmental Modeling and Software, September 2017, 95:320-325 DOI:10.1016/j.envsoft.2017.06.040
Kosko, B. (1986). Fuzzy Cognitive Maps. International Journal of Man-Machine Studies 24, pp. 65-75.
Perozo Niriaska, Jose Aguilar, Oswaldo Terán y Heidi Molina (2013). A Verification Method for MASOES, IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART B: CYBERNETICS, Vol 43, num 1, February, pp. 64-76, ISBN: 1083-4419. DOI: 10.1109/TSMCB.2012.2199106
Prensky, Marc (2001a). Digital Natives, Digital Immigrants Part 1, On the Horizon, Vol. 9, No. 5, September, pp 1-6. DOI: 10.1108/10748120110424816
Prensky, Marc (2001b). Do They Really Think Differently? Digital Natives, Digital Immigrants Part 2, On the Horizon, Vol. 9, No. 6, October 2001, pp 1-6. DOI: 10.1108/10748120110424843
Prensky, Marc (2009). “H. Sapiens Digital: From Digital Immigrants and Digital Natives to Digital Wisdom,” Innovate: Journal of Online Education: Vol. 5 : Issue 3, Article 1. Available at: https://nsuworks.nova.edu/innovate/vol5/iss3/1
Puerto E., Aguilar J., Chávez D., López C. (2019) Using Multilayer Fuzzy Cognitive Maps to Diagnose Autism Spectrum Disorder, Applied Soft Computing Journal, 75, pp. 58–71.
Rocco Paolillo, and Wander Jager (2020). Simulating Acculturation Dynamics Between Migrants and Locals in Relation to Network Formation, Social Science Computer Review. Volume 38 Issue 4, August 2020, pp. 365–386. https://journals.sagepub.com/toc/ssce/38/4
Sánchez Hebert, Jose Aguilar, Oswaldo Terán, José Gutiérrez de Mesa (2019). “Modeling the process of shaping the public opinion through Multilevel Fuzzy Cognitive Maps”, Applied Soft Computing, Volume 85. https://doi.org/10.1016/j.asoc.2019.105756.
Sibertin-Blanc, C., Roggero, P., Adreit, F., Baldet, B., Chapron, P., El-Gemayel, J., Mailliard, M., and Sandri, S. (2013). “SocLab: A Framework for the Modeling, Simulation and Analysis of Power in Social Organizations”, Journal of Artificial Societies and Social Simulation (JASSS), 16(4). http://jasss.soc.surrey.ac.uk/
Terán Oswaldo (2001). Emergent Tendencies in Multi-Agent Based Simulations Using Constraint-Based Methods to Effect Practical Proofs Over Finite Subsets of Simulation Outcomes, Doctoral Thesis, Centre for Policy Modelling, Manchester Metropolitan University, 2001.
Terán Oswaldo (2004). Understanding MABS and Social Simulation: Switching Between Languages in a Hierarchy of Levels, Journal of Artificial Societies and Social Simulation vol. 7, no. 4. http://jasss.soc.surrey.ac.uk/7/4/5.html
Terán Oswaldo, Edmonds Bruce and Steve Wallis (2001) “Mapping the Envelope of Social Simulation Trajectories”, Lecture Notes in Artificial Intelligence (Subserie de Lecture Notes in Computer Science): MABS, Volume 1979, pp. 229-243, Springer, Alemania, 2001
Terán Oswaldo y Jose Aguilar (2018). “Modelo del proceso de influencia de los medios de comunicación social en la opinión pública”, EDUCERE, 21 (71), Universidad de Los Andes, Mérida, Venezuela. https://www.redalyc.org/toc.oa?id=356&numero=56002
Terán Oswaldo and Christophe Sibertin-Blanc (2020). Impact on Cooperation of altruism, tenacity, and the need of a resource, accepted in IEEE Transactions on Computational Social Systems. DOI: 10.1109/TCSS.2021.3136821.
Annex. Fuzzy Cognitive Maps (FCM) and Muti-Fuzzy Cognitive Maps (MFCM).
Cognitive map theory is based on symbolic representation for the description of a system. It uses information, knowledge and experience, to describe particular domains using concepts (variables, states, inputs, outputs), and the relationships between them (Aguilar 2005, 2013, 2016). Cognitive maps can be understood as directed graphs, whose arcs represent causal connections between the nodes (concepts), used to denote knowledge. An arc with a positive sign (alternatively, negative sign), going from node X to node Y means that X (causally) increases (alternatively, decreases) Y. Cognitive maps are graphically represented: concepts are connected by arcs through a connection matrix. In the connection matrix, the i-nth line represents the weight of the arc connections directed outside of the concept. The i-nth column lists the arcs directed toward , i.e., those affecting .
The conceptual development of FCMs rests on the definition and dynamic of concepts and relationships created by the theory of fuzzy sets (Kosko, 1986). FCM can describe any system using a causality-based model (that indicates positive or negative relationships), which takes fuzzy values and is dynamic (i.e., the effect of a change in one concept/node affects other nodes, which then affect further nodes). This structure establishes the forward and backward propagation of causality (Aguilar, 2005, 2013, 2016). Thus, the concepts and relations can be represented as fuzzy variables (expressed in linguistic terms), such as “Almost Always”, “Always”, “Normally”, “Some (see Figure 2).
The value of a concept depends on its previous iterations, following the equation (1):
Cm(i+1) stands for the value of the concept in the next iteration after the iteration i, N indicates the number of concepts, wm,k represents the value of the causal relationship between the concept Ck and the concept Cm, and S(y) is a function used to normalize the value of the concept.
Figure 2. Example of an FCM (taken from Sánchez et al., 2019).
MFCM is an extension of the FCM. It is a FCM with several layers where each layer represents a set of concepts that define a specific domain of a system. To construct a MFCM, the previous equation for calculating the current status of the concepts of a FCM is modified, to describe the relationships between different layers (Aguilar, 2016):
Where F(p) is the input function generated by the relationships among different layers, and p is the set of concepts of the other layers that impact this concept. Thus, the update function of the concepts has two parts. The first part, the classic, calculates the value of concept in iteration based on the values of concepts in the previous iteration . All these concepts belong to the same layer where the “m” concept belongs. The second part is the result of the causal relationship between the concepts in different levels of the MFCM.
Terán, O. & Aguilar, J. (2022) Towards contextualized social simulation: Complementary use of Multi-Fuzzy Cognitive Maps and MABS. Review of Artificial Societies and Social Simulation, 25th May 2022. https://rofasss.org/2022/05/25/MFCM-MABS
By Peer-Olaf Siebers, in collaboration with Kwabena Amponsah, James Hey, Edmund Chattoe-Brown and Melania Borit
Motivation
1.1: Some time ago, I had several discussions with my PhD students Kwabena Amponsah and James Hey (we are all computer scientists, with a research interest in multi-agent systems) on the topic of qualitative vs. quantitative data in the context of Agent-Based Social Simulation (ABSS). Our original goal was to better understand the role of qualitative vs. quantitative data in the life cycle of an ABSS study. But as you will see later, we conquered more ground during our discussions.
1.2: The trigger for these discussions came from numerous earlier discussions within the RAT task force (Sebastian Achter, Melania Borit, Edmund Chattoe-Brown, and Peer-Olaf Siebers) on the topic, while we were developing the Rigour and Transparency – Reporting Standard (RAT-RS). The RAT-RS is a tool to improve the documentation of data use in Agent-Based Modelling (Achter et al 2022). During our RAT-RS discussions we made the observation that when using the terms “qualitative data” and “quantitative data” in different phases of the ABM simulation study life cycle these could be interpreted in different ways, and we felt difficult to clearly state the definition/role of these different types of data in the different contexts that the individual phases within the life cycle represent. This was aggravated by the competing understandings of the terminology within different domains (from social and natural sciences) that form the field of social simulation.
1.3: As the ABSS community is a multi-disciplinary one, often doing interdisciplinary research, we thought that we should share the outcome of our discussions with the community. To demonstrate the different views that exist within the topic area, we ask some of our friends from the social simulation community to comment on our philosophical discussions. And we were lucky enough to get our RAT-RS colleagues Edmund Chattoe-Brown and Melania Borit on board who provided critical feedback and their own view of things. In the following we provide a summary of the overall discussion Each of the following paragraph contains summaries of the initial discussion outcomes, representing the computer scientists’ views, followed by some thoughts provided by our two friends from the social simulation community (Borit’s in {} brackets and italicand Chattoe-Brown’s in [] brackets and bold), both commenting on the initial discussion outcomes of the computer scientists. To see the diverse backgrounds of all the contributors and perhaps to better understand their way of thinking and their arguments, I have added some short biographies of all contributors at the end of this Research Note. To support further (public) discussions I have numbered the individual paragraphs to make it easier to refer back to them.
Terminology
2.1: As a starting point for our discussions I searched the internet for some terminology related to the topic of “data”. Following is a list of initial definitions of relevant terms [1]. First, the terms qualitative data and quantitative data, as defined by the Australian Bureau of Statistics: “Qualitative data are measures of ‘types’ and may be represented by a name, symbol, or a number code. They are data about categorical variables (e.g. what type). Quantitative data are measures of values or counts and are expressed as numbers. They are data about numeric variables (e.g. how many; how much; or how often).” (Australian Bureau of Statistics 2022) [Maybe don’t let a statistics unit define qualitative research? This has a topic that is very alien to us but argues “properly” about the role of different methods (Helitzer-Allen and Kendall 1992). “Proper” qualitative researchers would fiercely dispute this. It is “quantitative imperialism”.].
2.2: What might also help for this discussion is to better understand the terms qualitative data analysis and quantitative data analysis. Qualitative data analysis refers to “the processes and procedures that are used to analyse the data and provide some level of explanation, understanding, or interpretation” (Skinner et al 2021). [This is a much less contentious claim for qualitative data – and makes the discussion of the Australian Bureau of Statistics look like a distraction but a really “low grade” source in peer review terms. A very good one is Strauss (1987).] These methods include content analysis, narrative analysis, discourse analysis, framework analysis, and grounded theory and the goal is to identify common patterns. {These data analysis methods connect to different types of qualitative research: phenomenology, ethnography, narrative inquiry, case study research, or grounded theory. The goal of such research is not always to identify patterns – see (Miles and Huberman 1994): e.g., making metaphors, seeing plausibility, making contrasts/comparisons.}[In my opinion some of these alleged methods are just empire building or hot air. Do you actually need them for your argument?] These types of analysis must therefore use qualitative inputs, broadening the definition to include raw text, discourse and conceptual frameworks.
2.3 When it comes to quantitative data analysis “you are expected to turn raw numbers into meaningful data through the application of rational and critical thinking. Quantitative data analysis may include the calculation of frequencies of variables and differences between variables.” (Business Research Methodology 2022) {One does the same in qualitative data analysis – turns raw words (or pictures etc.) into meaningful data through the application of interpretation based on rational and critical thinking. In quantitative data analysis you usually apply mathematical/statistical models to analyse the data.}. While the output of quantitative data analysis can be used directly as input to a simulation model, the output of qualitative data analysis still needs to be translated into behavioural rules to be useful (either manually or through machine learning algorithms). {What is meant by “translated” in this specific context? Do we need this kind of translation only for qualitative data or also for quantitative data? Is there a difference between translation methods of qualitative and quantitative data?} [That seems pretty contentious too. It is what is often done, true, but I don’t think it is a logical requirement. I guess you could train a neural net using “cases” or design some other simple “cognitive architecture” from the data. Would this (Becker 1953), for example, best be modelled as “rules” or as some kind of adaptive process? But of course you have to be careful that “rule” is not defined so broadly that everything is one or it is true by definition. I wonder what the “rules” are in this: Chattoe-Brown (2009).]
2.4: Finally, let’s have a quick look at the difference between “data” and “evidence”. For this, we found the following distinction by Wilkinson (2022) helpful: “… whilst data can exist on its own, even though it is essentially meaningless without context, evidence, on the other hand, has to be evidence of or for something. Evidence only exists when there is an opinion, a viewpoint or an argument.”
Hypothesis
3.1: The RAT-RS divides the simulation life cycle into five phases, in terms of data use: model aim and context, conceptualisation, operationalisation, experimentation, and evaluation (Siebers et al 2019). We started our discussion by considering the following hypothesis: The outcome of qualitative data analysis is only useful for the purpose of conceptualisation and as a basis for producing quantitative data. It does not have any other roles within the ABM simulation study life cycle. {Maybe this hypothesis in itself has to be discussed. Is it so that you use only numbers in the operationalisation phase? One can write NetLogo code directly from qualitative data, without numbers.} [Is this inevitable given the way ABM works? Agents have agency and therefore can decide to do things (and we can only access this by talking to them, probably “open ended”). A statistical pattern – time series or correlation – has no agency and therefore cannot be accessed “qualitatively” – though we also sometimes mean by “qualitative” eyeballing two time series rather than using some formal measure of tracking. I guess that use would >>not<< be relevant here.]
Discussion
4.1: One could argue that qualitative data analysis provides causes for behaviour (and indications about their importance (ranking); perhaps also the likelihood of occurrence) as well as key themes that are important to be considered in a model. All sounds very useful for the conceptual modelling phase. The difficulty might be to model the impact (how do we know we model it correctly and at the right level), if that is not easily translatable into a quantitative value but requires some more (behavioural) mechanistic structures to represent the impact of behaviours. [And, of course, there is a debate in psychology (with some evidence on both sides) about the extent to which people are able to give subjective accounts we can trust (see Hastorf and Cantril (1954).] This might also provide issues when it comes to calibration – how does one calibrate qualitative data? {Triangulation.} One random idea we had was that perhaps fuzzy logic could help with this. More brainstorming and internet research is required to confirm that this idea is feasible and useful. [A more challenging example might be ethnographic observation of a “neighbourhood” in understanding crime. This is not about accessing the cognitive content of agents but may well still contribute to a well specified model. It is interesting how many famous models – Schelling, Zaller-Deffuant – actually have no “real” environment.]
4.2: One could also argue that what starts (or what we refer to initially) as qualitative data always ends up as quantitative data, as whatever comes out of the computer are numbers. {This is not necessarily true. Check the work on qualitative outputs using Grounded Theory by Neumann (2015).} Of course this is a question related to the conceptual viewpoint. [Not convinced. It sounds like all sociology is actually physics because all people are really atoms. Formally, everything in computers is numbers because it has to be but that isn’t the same as saying that data structures or whatever don’t constitute a usable and coherent level of description: We “meet” and “his” opinion changes “mine” and vice versa. Somewhere, that is all binary but you can read the higher level code that you can understand as “social influence” (whatever you may think of the assumptions). Be clear whether this (like the “rules” claim) is a matter of definition – in which case it may not be useful (even if people are atoms we have no idea of how to solve the “atomic physics” behind the Prisoner’s Dilemma) or an empirical one (in which case some models may just prove it false). This (Beltratti et al 1996) contains no “rules” and no “numbers” (except in the trivial sense that all programming does).]
4.3: Also, an algorithm is expressed in code and can only be processed numerically, so it can only deliver quantitative data as output. These can perhaps be translated into qualitative concepts later. A way of doing this via the use of grounded theory is proposed in Neumann and Lotzmann (2016). {This refers to the same idea as my previous comment.} [Maybe it is “safest” to discuss this with rules because everyone knows those are used in ABM. Would it make sense to describe the outcome of a non trivial set of rules – accessed for example like this: Gladwin (1989) – as either “quantitative” or “numbers?”]
4.4: But is it true that data delivered as output is always quantitative? Let’s consider, for example, a consumer marketing scenario, where we define stereotypes (shopping enthusiast; solution demander; service seeker; disinterested shopper; internet shopper) that can change over time during a simulation run (Siebers et al 2010). These stereotypes are defined by likelihoods (likelihood to buy, wait, ask for help, and ask for refund). So, during a simulation run an agent could change its stereotype (e.g. from shopping enthusiast to disinterested shopper), influenced by the opinion of others and their own previous experience. So, at the beginning of the simulation run the agent can have a different stereotype compared to the end. Of course we could enumerate the five different stereotypes, and claim that the outcome is numeric, but the meaning of the outcome would be something qualitative – the stereotype related to that number. To me this would be a qualitative outcome, while the number of people that change from one stereotype to another would be a quantitative outcome. They would come in a tandem. {So, maybe the problem is that we don’t yet have the right ways of expressing or visualising qualitative output?} [This is an interesting and grounded example but could it be easily knocked down because everything is “hard coded” and therefore quantifiable? You may go from one shopper type to another – and what happens depends on other assumptions about social influence and so on – but you can’t “invent” your own type. Compare something like El Farol (Arthur 1994) where agents arguably really can “invent” unique strategies (though I grant these are limited to being expressed in a specified “grammar”).]
4.5: In order to define someone’s stereotype we would use numerical values (likelihood = proportion). However, stereotypes refer to nominal data (which refers to data that is used for naming or labelling variables, without any quantitative value). The stereotype itself would be nominal, while the way one would derive the stereotype would be numerical. Figure 1 illustrates a case in which the agent moves from the disinterested stereotype to the enthusiast stereotype. [Is there a potential confusion here between how you tell an agent is a type – parameters in the code just say so – and how you say a real person is a type? Everything you say about the code still sounds “quantitative” because all the “ingredients” are.]
Figure 1: Enthusiastic and Disinterested agent stereotypes
4.6: Let’s consider a second example, related to the same scenario: The dynamics over time to get from an enthusiastic shopper (perhaps via phases) to a disinterested shopper. This is represented as a graph where the x-axis represents time and the y-axis stereotypes (categorical data). If you want to take a quantitative perspective on the outcome you would look at a specific point in time (state of the system) but to take a qualitative perspective of the outcome, you would look at the pattern that the curve represents over the entire simulation runtime. [Although does this shade into the “eyeballing” sense of qualitative rather than the “built from subjective accounts” sense? Another way to think of this issue is to imagine “experts” as a source of data. We might build an ABM based on an expert perception of say, how a crime gang operates. That would be qualitative but not just individual rules: For example, if someone challenges the boss to a fight and loses they die or leave. This means the boss often has no competent potential successors.]
4.7: So, the inputs (parameters, attributes) to get the outcome are numeric, but the outcome itself in the latter case is not. The outcome only makes sense once it’s put into the qualitative context. And then we could say that the simulation produces some qualitative outputs. So, does the fact that data needs to be seen in a context make it evidence, i.e. do we only have quantitative and qualitative evidence on the output side? [Still worried that you may not be happy equating qualitative interview data with qualitative eyeballing of graphs. Mixes up data collection and analysis? And unlike qualitative interviews you don’t have to eyeball time series. But the argument of qualitative research is you can’t find out some things any other way because, to run a survey say, or an experiment, you already have to have a pretty good grasp of the phenomenon.]
4.8: If one runs a marketing campaign that will increase the number of enthusiastic shoppers. This can be seen as qualitative data as it is descriptive of how the system works rather than providing specific values describing the performance of a system. You could also equally express this algebraic terms which would make it quantitative data. So, it might be useful to categorise quantitative data to make the outcome easier to understand. [I don’t think this argument is definitely wrong – though I think it may be ambiguous about what “qualitative” means – but I think it really needs stripping down and tightening. I’m not completely convinced as a new reader that I’m getting at the nub of the argument. Maybe just one example in detail and not two in passing?]
Outcome
5.1: How we understand things and how the computer processes things are two different things. So, in fact qualitative data is useful for the conceptualisation and for describing experimentation and evaluation output, and needs to be translated into numerical data or algebraic constructs for the operationalisation. Therefore, we can reject our initial hypothesis, as we found more places where qualitative data can be useful. [Yes, and that might form the basis for a “general” definition of qualitative that was not tied to one part of the research process but you would have to be clear that’s what you were aiming at and not just accidentally blurring two different “senses” of qualitative.]
5.2: In the end of the discussion we picked up the idea of using Fuzzy Logic. Could perhaps fuzzy logic be used to describe qualitative output, as it describes a degree of membership to different categories? An interesting paper to look at in this context would be Sugeno and Yasukawa (1993). Also, a random idea that was mentioned is if there is potential in using “fuzzy logic in reverse”, i.e. taking something that is fuzzy, making it crisp for the simulation, and making it fuzzy again for presenting the result. However, we decided to save this topic for another discussion. [Devil will be in the detail. Depends on exactly what assumptions the method makes. Devil’s advocate: What if qualitative research is only needed for specification – not calibration or validation – but it doesn’t follow from this that that use is “really” quantitative? How intellectually unappealing is that situation and why?]
Conclusion
6.1: The purpose of this Research Note is really to stimulate you to think about, talk about, and share your ideas and opinions on the topic! What we present here is a philosophical impromptu discussion of our individual understanding of the topic, rather than a scientific debate that is backed up by literature. We still thought it is worthwhile to share this with you, as you might stumble across similar questions. Also, we don’t think we have found the perfect answers to the questions yet. So we would like to invite you to join the discussion and leave some comments in the chat, stating your point of view on this topic. [Is the danger of discussing these data types “philosophically”? I don’t know if it is realistic to use examples directly from social simulation but for sure examples can be used from social science generally. So here is a “quantitative” argument from quantitative data: “The view that cultural capital is transmitted from parents to their children is strongly supported in the case of pupils’ cultural activities. This component of pupils’ cultural capital varies by social class, but this variation is entirely mediated by parental cultural capital.” (Sullivan 2001). As well as the obvious “numbers” (social class by a generally agreed scheme) there is also a constructed “measure” of cultural capital based on questions like “how many books do you read in a week?” Here is an example of qualitative data from which you might reason: “I might not get into Westbury cos it’s siblings and how far away you live and I haven’t got any siblings there and I live a little way out so I might have to go on a waiting list … I might go to Sutton Boys’ instead cos all my mates are going there.” (excerpt from Reay 2002). As long as this was not just a unique response (but was supported by several other interviews) one would add to one’s “theory” of school choice: 1) Awareness of the impact of the selection system (there is no point in applying here whatever I may want) and 2) The role of networks in choice: This might be the best school for me educationally but I won’t go because I will be lonely.]
Biographies of the authors
Peer-Olaf Siebers is an Assistant Professor at the School of Computer Science, University of Nottingham, UK. His main research interest is the application of Computer Simulation and Artificial Intelligence to study human-centric and coupled human-natural systems. He is a strong advocate of Object-Oriented Agent-Based Social Simulation and is advancing the methodological foundations. It is a novel and highly interdisciplinary research field, involving disciplines like Social Science, Economics, Psychology, Geography, Operations Research, and Computer Science.
Kwabena Amponsah is a Research Software Engineer working for the Digital Research Service, University of Nottingham, UK. He completed his PhD in Computer Science at Nottingham in 2019 by developing a framework for evaluating the impact of communication on performance in large-scale distributed urban simulations.
James Hey is a PhD student at the School of Computer Science, University of Nottingham, UK. In his PhD he investigates the topic of surrogate optimisation for resource intensive agent based simulation of domestic energy retrofit uptake with environmentally conscious agents. James holds a Bachelor degree in Economics as well as a Master degree in Computer Science.
Edmund Chattoe-Brown is a lecturer in Sociology, School of Media, Communication and Sociology, University of Leicester, UK. His career has been interdisciplinary (including Politics, Philosophy, Economics, Artificial Intelligence, Medicine, Law and Anthropology), focusing on the value of research methods (particularly Agent-Based Modelling) in generating warranted social knowledge. His aim has been to make models both more usable generally and particularly more empirical (because the most rigorous social scientists tend to be empirical). The results of his interests have been published in 17 different peer reviewed journals across the sciences to date. He was funded by the project “Towards Realistic Computational Models Of Social Influence Dynamics” (ES/S015159/1) by the ESRC via ORA Round 5.
Melania Borit is an interdisciplinary researcher and the leader of the CRAFT Lab – Knowledge Integration and Blue Futures at UiT The Arctic University of Norway. She has a passion for knowledge integration and a wide range of interconnected research interests: social simulation, agent-based modelling; research methodology; Artificial Intelligence ethics; pedagogy and didactics in higher education, games and game-based learning; culture and fisheries management, seafood traceability; critical futures studies.
References
Achter S, Borit M, Chattoe-Brown E, and Siebers PO (2022) RAT-RS: a reporting standard for improving the documentation of data use in agent-based modelling. International Journal of Social Research Methodology, DOI: 10.1080/13645579.2022.2049511
Chattoe-Brown E (2009) The social transmission of choice: a simulation with applications to hegemonic discourse. Mind & Society, 8(2), pp.193-207. DOI: 10.1007/s11299-009-0060-7
Gladwin CH (1989) Ethnographic Decision Tree Modeling. SAGE Publications.
Hastorf AH and Cantril H (1954) They saw a game; a case study. The Journal of Abnormal and Social Psychology, 49(1), pp.129–134.
Helitzer-Allen DL and Kendall C (1992) Explaining differences between qualitative and quantitative data: a study of chemoprophylaxis during pregnancy. Health Education Quarterly, 19(1), pp.41-54. DOI: 10.1177%2F109019819201900104
Miles MB and Huberman AM (1994) . Qualitative Data Analysis: An Expanded Sourcebook. Sage
Neumann M (2015) Grounded simulation. Journal of Artificial Societies and Social Simulation, 18(1)9. DOI: 10.18564/jasss.2560
Neumann M and Lotzmann U (2016) Simulation and interpretation: a research note on utilizing qualitative research in agent based simulation. International Journal of Swarm Intelligence and Evolutionary Computing 5/1.
Reay D (2002) Shaun’s Story: Troubling discourses of white working-class masculinities. Gender and Education, 14(3), pp.221-234. DOI: 10.1080/0954025022000010695
Siebers PO, Achter S, Palaretti Bernardo C, Borit M, and Chattoe-Brown E (2019) First steps towards RAT: a protocol for documenting data use in the agent-based modeling process (Extended Abstract). Social Simulation Conference 2019 (SSC 2019), 23-27 Sep, Mainz, Germany.
Siebers PO, Aickelin U, Celia H and Clegg C (2010) Simulating customer experience and word-of-mouth in retail: a case study. Simulation: Transactions of the Society for Modeling and Simulation International, 86(1) pp. 5-30. DOI: 10.1177%2F0037549708101575
Skinner J, Edwards A and Smith AC (2021) Qualitative Research in Sport Management – 2e, p171. Routledge.
Strauss AL (1987). Qualitative Analysis for Social Scientists. Cambridge University Press.
Sugeno M and Yasukawa T (1993) A fuzzy-logic-based approach to qualitative modeling. IEEE Transactions on Fuzzy Systems, 1(1), pp.7-31.
Sullivan A (2001) Cultural capital and educational attainment. Sociology 35(4), pp.893-912. DOI: 10.1017/S0038038501008938
[1] An updated set of the terminology, defined by the RAT task force in 2022, is available as part of the RAT-RS in Achter et al (2022) Appendix A1.
Peer-Olaf Siebers, Kwabena Amponsah, James Hey, Edmund Chattoe-Brown and Melania Borit (2022) Discussions on Qualitative & Quantitative Data in the Context of Agent-Based Social Simulation. Review of Artificial Societies and Social Simulation, 16th May 2022. https://rofasss.org/2022/05/16/Q&Q-data-in-ABM
1Department of Computing Science, Faculty of Science and Technology, Umeå University, frank.dignum@umu.se 2Centre for Policy Modelling, Manchester Metropolitan University, bruce@edmonds.name 3Department of Psychology, University of Limerick, dino.carpentras@gmail.com
In this position paper we argue for the creation of a new ‘field’: Socio-Cognitive Systems. The point of doing this is to highlight the importance of a multi-levelled approach to understanding those phenomena where the cognitive and the social are inextricably intertwined – understanding them together.
What goes on ‘in the head’ and what goes on ‘in society’ are complex questions. Each of these deserves serious study on their own – motivating whole fields to answer them. However, it is becoming increasingly clear that these two questions are deeply related. Humans are fundamentally social beings, and it is likely that many features of their cognition have evolved because they enable them to live within groups (Herrmann et al. 20007). Whilst some of these social features can be studied separately (e.g. in a laboratory), others only become fully manifest within society at large. On the other hand, it is also clear that how society ‘happens’ is complicated and subtle and that these processes are shaped by the nature of our cognition. In other words, what people ‘think’ matters for understanding how society ‘is’ and vice versa. For many reasons, both of these questions are difficult to answer. As a result of these difficulties, many compromises are necessary in order to make progress on them, but each compromise also implies some limitations. The main two types of compromise consist of limiting the analysis to only one of the two (i.e. either cognition or society)[1]. To take but a few examples of this.
Neuro-scientists study what happens between systems of neurones to understand how the brain does things and this is so complex that even relatively small ensembles of neurones are at the limits of scientific understanding.
Psychologists see what can be understood of cognition from the outside, usually in the laboratory so that some of the many dimensions can be controlled and isolated. However, what can be reproduced in a laboratory is a limited part of behaviour that might be displayed in a natural social context.
Economists limit themselves to the study of the (largely monetary) exchange of services/things that could occur under assumptions of individual rationality, which is a model of thinking not based upon empirical data at the individual level. Indeed it is known to contradict a lot of the data and may only be a good approximation for average behaviour under very special circumstances.
Ethnomethodologists will enter a social context and describe in detail the social and individual experience there, but not generalise beyond that and not delve into the cognition of those they observe.
Other social scientists will take a broader view, look at a variety of social evidence, and theorise about aspects of that part of society. They (almost always) do not include individual cognition into account in these and do not seek to integrate the social and the cognitive levels.
Each of these in the different ways separate the internal mechanisms of thought from the wider mechanisms of society or limits its focus to a very specific topic. This is understandable; what each is studying is enough to keep them occupied for many lifetimes. However, this means that each of these has developed their own terms, issues, approaches and techniques which make relating results between fields difficult (as Kuhn, 1962, pointed out).
Figure 1: Schematic representation of the relationship between the individual and society. Individuals’ cognition is shaped by society, at the same time, society is shaped by individuals’ beliefs and behaviour.
This separation of the cognitive and the social may get in the way of understanding many things that we observe. Some phenomena seem to involve a combination of these aspects in a fundamental way – the individual (and its cognition) being part of society as well as society being part of the individual. Some examples of this are as follows (but please note that this is far from an exhaustive list).
Norms. A social norm is a constraint or obligation upon action imposed by society (or perceived as such). One may well be mistaken about a norm (e.g. whether it is ok to casually talk to others at a bus stop), thus it is also a belief – often not told to one explicitly but something one needs to infer from observation. However, for a social norm to hold it also needs to be an observable convention. Decisions to violate social norms require that the norm is an explicit (referable) object in the cognitive model. But the violation also has social consequences. If people react negatively to violations the norm can be reinforced. But if violations are ignored it might lead to a norm disappearing. How new norms come about, or how old ones fade away, is a complex set of interlocking cognitive and social processes. Thus social norms are a phenomena that essentially involves both the social and the cognitive (Conte et al. 2013).
Joint construction of social reality. Many of the constraints on our behaviour come from our perception of social reality. However, we also create this social reality and constantly update it. For example, we can invent a new procedure to select a person as head of department or exit a treaty and thus have different ways of behaving after this change. However, these changes are not unconstrained in themselves. Sometimes the time is “ripe for change”, while at other times resistance is too big for any change to take place (even though a majority of the people involved would like to change). Thus what is socially real for us depends on what people individually believe is real, but this depends in complex ways on what other people believe and their status. And probably even more important: the “strength” of a social structure depends on the use people make of it. E.g. a head of department becomes important if all decisions in the department are deferred to the head. Even though this might not be required by university or law.
Identity. Our (social) identity determines the way other people perceive us (e.g. a sports person, a nerd, a family man) and therefore creates expectations about our behaviour. We can create our identities ourselves and cultivate them, but at the same time, when we have a social identity, we try to live up to it. Thus, it will partially determine our goals and reactions and even our feeling of self-esteem when we live up to our identity or fail to do so. As individuals we (at least sometimes) have a choice as to our desired identity, but in practice, this can only be realised with the consent of society. As a runner I might feel the need to run at least three times a week in order for other people to recognize me as runner. At the same time a person known as a runner might be excused from a meeting if training for an important event. Thus reinforcing the importance of the “runner” identity.
Social practices. The concept already indicates that social practices are about the way people habitually interact and through this interaction shape social structures. Practices like shaking hands when greeting do not always have to be efficient, but they are extremely socially important. For example, different groups, countries and cultures will have different practices when greeting and performing according to the practice shows whether you are part of the in-group or out-group. However, practices can also change based on circumstances and people, as it happened, for example, to the practice of shaking hands during the covid-19 pandemic. Thus, they are flexible and adapting to the context. They are used as flexible mechanisms to efficiently fit interactions in groups, connecting persons and group behaviour.
As a result, this division between cognitive and the social gets in the way not only of theoretical studies, but also in practical applications such as policy making. For example, interventions aimed at encouraging vaccination (such as compulsory vaccination) may reinforce the (social) identity of the vaccine hesitant. However, this risk and its possible consequences for society cannot be properly understood without a clear grasp of the dynamic evolution of social identity.
Computational models and systems provide a way of trying to understand the cognitive and the social together. For computational modellers, there is no particular reason to confine themselves to only the cognitive or only the social because agent-based systems can include both within a single framework. In addition, the computational system is a dynamic model that can represent the interactions of the individuals that connect the cognitive models and the social models. Thus the fact that computational models have a natural way to represent the actions as an integral and defining part of the socio-cognitive system is of prime importance. Given that the actions are an integral part of the model it is well suited to model the dynamics of socio-cognitive systems and track changes at both the social and the cognitive level. Therefore, within such systems we can study how cognitive processes may act to produce social phenomena whilst, at the same time, as how social realities are shaping the cognitive processes. Caarley and Newell (1994) discusses what is necessary at the agent level for sociality, Hofested et al. (2021) talk about how to understand sociality using computational models (including theories of individual action) – we want to understand both together. Thus, we can model the social embeddedness that Granovetter (1985) talked about – going beyond over- or under-socialised representations of human behaviour. It is not that computational models are innately suitable for modelling either the cognitive or the social, but that they can be appropriately structured (e.g. sets of interacting parts bridging micro-, meso- and macro-levels) and include arbitrary levels of complexity. Lots of models that represent the social have entities that stand for the cognitive, but do not explicitly represent much of that detail – similarly much cognitive modelling implies the social in terms of the stimuli and responses of an individual that would be to other social entities, but where these other entities are not explicitly represented or are simplified away.
Socio-Cognitive Systems (SCS) are: those models and systems where both cognitive and social complexity are represented with a meaningful level of processual detail.
A good example of an application where this appeared of the biggest importance was in simulations for the covid-19 crisis. The spread of the corona virus on macro level could be given by an epidemiological model, but the actual spreading depended crucially on the human behaviour that resulted from individuals’ cognitive model of the situation. In Dignum (2021) it was shown how the socio-cognitive system approach was fundamental to obtaining better insights in the effectiveness of a range of covid-19 restrictions.
Formality here is important. Computational systems are formal in the sense that they can be unambiguously passed around (i.e. unlike language, it is not differently re-interpreted by each individual) and operate according to their own precisely specified and explicit rules. This means that the same system can be examined and experimented on by a wider community of researchers. Sometimes, even when the researchers from different fields find it difficult to talk to one another, they can fruitfully cooperate via a computational model (e.g. Lafuerza et al. 2016). Other kinds of formal systems (e.g. logic, maths) are geared towards models that describe an entire system from a birds eye view. Although there are some exceptions like fibred logics Gabbay (1996), these are too abstract to be of good use to model practical situations. The lack of modularity and has been addressed in context logics Giunchiglia, F., & Ghidini, C. (1998). However, the contexts used in this setting are not suitable to generate a more general societal model. It results in most typical mathematical models using a number of agents which is either one, two or infinite (Miller and Page 2007), while important social phenomena happen with a “medium sized” population. What all these formalisms miss is a natural way of specifying the dynamics of the system that is modelled, while having ways to modularly describe individuals and the society resulting from their interactions. Thus, although much of what is represented in Socio-Cognitive Systems is not computational, the lingua franca for talking about them is.
The ‘double complexity’ of combining the cognitive and the social in the same system will bring its own methodological challenges. Such complexity will mean that many socio-cognitive systems will be, themselves, hard to understand or analyse. In the covid-19 simulations, described in (Dignum 2021), a large part of the work consisted of analysing, combining and representing the results in ways that were understandable. As an example, for one scenario 79 pages of graphs were produced showing different relations between potentially relevant variables. New tools and approaches will need to be developed to deal with this. We only have some hints of these, but it seems likely that secondary stages of analysis – understanding the models – will be necessary, resulting in a staged approach to abstraction (Lafuerza et al. 2016). In other words, we will need to model the socio-cognitive systems, maybe in terms of further (but simpler) socio-cognitive systems, but also maybe with a variety of other tools. We do not have a view on this further analysis, but this could include: machine learning, mathematics, logic, network analysis, statistics, and even qualitative approaches such as discourse analysis.
An interesting input for the methodology of designing and analysing socio-cognitive systems is anthropology and specifically ethnographical methods. Again, for the covid-19 simulations the first layer of the simulation was constructed based on “normal day life patterns”. Different types of persons were distinguished that each have their own pattern of living. These patterns interlock and form a fabric of social interactions that overall should satisfy most of the needs of the agents. Thus we calibrate the simulation based on the stories of types of people and their behaviours. Note that doing the same just based on available data of behaviour would not account for the underlying needs and motives of that behaviour and would not be a good basis for simulating changes. The stories that we used looked very similar to the type of reports ethnographers produce about certain communities. Thus further investigating this connection seems worthwhile.
For representing the output of the complex socio-cognitive systems we can also use the analogue of stories. Basically, different stories show the underlying (assumed) causal relations between phenomena that are observed. E.g. seeing an increase in people having lunch with friends can be explained by the fact that a curfew prevents people having dinner with their friends, while they still have a need to socialize. Thus the alternative of going for lunch is chosen more often. One can see that the explaining story uses both social as well as cognitive elements to describe the results. Although in the covid-19 simulations we have created a number of these stories, they were all created by hand after (sometimes weeks) of careful analysis of the results. Thus for this kind of approach to be viable, new tools are required.
Although human society is the archetypal socio-cognitive system, it is not the only one. Both social animals and some artificial systems also come under this category. These may be very different from the human, and in the case of artificial systems completely different. Thus, Socio-Cognitive Systems is not limited to the discussion of observable phenomena, but can include constructed or evolved computational systems, and artificial societies. Examination of these (either theoretically or experimentally) opens up the possibility of finding either contrasts or commonalities between such systems – beyond what happens to exist in the natural world. However, we expect that ideas and theories that were conceived with human socio-cognitive systems in mind might often be an accessible starting point for understanding these other possibilities.
In a way, Socio-Cognitive Systems bring together two different threads in the work of Herbert Simon. Firstly, as in Simon (1948) it seeks to take seriously the complexity of human social behaviour without reducing this to overly simplistic theories of individual behaviour. Secondly, it adopts the approach of explicitly modelling the cognitive in computational models (Newell & Simon 1972). Simon did not bring these together in his lifetime, perhaps due to the limitations and difficulty of deploying the computational tools to do so. Instead, he tried to develop alternative mathematical models of aspects of thought (Simon 1957). However, those models were limited by being mathematical rather than computational.
To conclude, a field of Socio-Cognitive Systems would consider the cognitive and the social in an integrated fashion – understanding them together. We suggest that computational representation or implementation might be necessary to provide concrete reference between the various disciplines that are needed to understand them. We want to encourage research that considers the cognitive and the social in a truly integrated fashion. If by labelling a new field does this it will have achieved its purpose. However, there is the possibility that completely new classes of theory and complexity may be out there to be discovered – phenomena that are denied if either the cognitive or the social are not taken together – a new world of a socio-cognitive systems.
Notes
[1] Some economic models claim to bridge between individual behaviour and macro outcomes, however this is traditionally notional. Many economists admit that their primary cognitive models (varieties of economic rationality) are not valid for individuals but are what people on average do – i.e. this is a macro-level model. In other economic models whole populations are formalised using a single representative agent. Recently, there are some agent-based economic models emerging, but often limited to agree with traditional models.
Acknowledgements
Bruce Edmonds is supported as part of the ESRC-funded, UK part of the “ToRealSim” project, grant number ES/S015159/1.
References
Carley, K., & Newell, A. (1994). The nature of the social agent. Journal of mathematical sociology, 19(4): 221-262. DOI: 10.1080/0022250X.1994.9990145
Conte R., Andrighetto G. and Campennì M. (eds) (2013) Minding Norms – Mechanisms and dynamics of social order in agent societies. Oxford University Press, Oxford.
Dignum, F. (ed.) (2021) Social Simulation for a Crisis; Results and Lessons from Simulating the COVID-19 Crisis. Springer.
Herrmann E., Call J, Hernández-Lloreda MV, Hare B, Tomasello M (2007) Humans have evolved specialized skills of social cognition: The cultural intelligence hypothesis. Science317(5843): 1360-1366. DOI: 10.1126/science.1146282
Hofstede, G.J, Frantz, C., Hoey, J., Scholz, G. and Schröder, T. (2021) Artificial Sociality Manifesto. Review of Artificial Societies and Social Simulation, 8th Apr 2021. https://rofasss.org/2021/04/08/artsocmanif/
Gabbay, D. M. (1996). Fibred Semantics and the Weaving of Logics Part 1: Modal and Intuitionistic Logics. The Journal of Symbolic Logic, 61(4), 1057–1120.
Ghidini, C., & Giunchiglia, F. (2001). Local models semantics, or contextual reasoning= locality+ compatibility. Artificial intelligence, 127(2), 221-259. DOI: 10.1016/S0004-3702(01)00064-9
Granovetter, M. (1985) Economic action and social structure: The problem of embeddedness. American Journal of Sociology91(3): 481-510. DOI: 10.1086/228311
Kuhn, T,S, (1962) The structure of scientific revolutions. University of Chicago Press, Chicago
Miller, J. H., Page, S. E., & Page, S. (2009). Complex adaptive systems. Princeton university press.
Newell A, Simon H.A. (1972) Human problem solving. Prentice Hall, Englewood Cliffs, NJ
Simon, H.A. (1948) Administrative behaviour: A study of the decision making processes in administrative organisation. Macmillan, New York
Simon, H.A. (1957) Models of Man: Social and rational. John Wiley, New York
Dignum, F., Edmonds, B. and Carpentras, D. (2022) Socio-Cognitive Systems – A Position Statement. Review of Artificial Societies and Social Simulation, 2nd Apr 2022. https://rofasss.org/2022/04/02/scs
By Dino Carpentras1, Edmund Chattoe-Brown2*, Bruce Edmonds3, Cesar García-Diaz4, Christian Kammler5, Anna Pagani6 and Nanda Wijermans7
*Corresponding author, 1Centre for Social Issues Research, University of Limerick, 2School of Media, Communication and Sociology, University of Leicester, 3Centre for Policy Modelling, Manchester Metropolitan University, 4Department of Business Administration, Pontificia Universidad Javeriana, 5Department of Computing Science, Umeå University, 6Laboratory on Human-Environment Relations in Urban Systems (HERUS), École Polytechnique Fédérale de Lausanne (EPFL), 7Stockholm Resilience Centre, Stockholm University.
Introduction
This round table was convened to advance and improve the use of experimental methods in Agent-Based Modelling, in the hope that both existing and potential users of the method would be able to identify steps towards this aim[i]. The session began with a presentation by Bruce Edmonds (http://cfpm.org/slides/experiments%20and%20ABM.pptx) whose main argument was that the traditional idea of experimentation (controlling extensively for the environment and manipulating variables) was too simplistic to add much to the understanding of the sort of complex systems modelled by ABMs and that we should therefore aim to enhance experiments (for example using richer experimental settings, richer measures of those settings and richer data – like discussions between participants as well as their behaviour). What follows is a summary of the main ideas discussed organised into themed sections.
What Experiments Are
Defining the field of experiments proved to be challenging on two counts. The first was that there are a number of labels for potentially relevant approaches (experiments themselves – for example, Boero et al. 2010, gaming – for example, Tykhonov et al. 2008, serious games – for example Taillandier et al. 2019, companion/participatory modelling – for example, Ramanath and Gilbert 2004 and web based gaming – for example, Basole et al. 2013) whose actual content overlap is unclear. Is it the case that a gaming approach is generally more in line with the argument proposed by Edmonds? How can we systematically distinguish the experimental content of a serious game approach from a gaming approach? This seems to be a problem in immature fields where the labels are invented first (often on the basis of a few rather divergent instances) and the methodology has to grow into them. It would be ludicrous if we couldn’t be sure whether a piece of research was survey based or interview based (and this would radically devalue the associated labels if it were so.)
The second challenge is also more general in Agent-Based Modelling which is the same labels being used differently by different researchers. It is not productive to argue about which uses are correct but it is important that the concepts behind the different uses are clear so a common scheme of labelling might ultimately be agreed. So, for example, experiment can be used (and different round table participants had different perspectives on the uses they expected) to mean laboratory experiments (simplified settings with human subjects – again see, for example, Boero et al. 2010), experiments with ABMs (formal experimentation with a model that doesn’t necessarily have any empirical content – for example, Doran 1998) and natural experiments (choice of cases in the real world to, for example, test a theory – see Dinesen 2013).
One approach that may help with this diversity is to start developing possible dimensions of experimentation. One might be degree of control (all the way from very stripped down behavioural laboratory experiments to natural situations where the only control is to select the cases). Another might be data diversity: From pure analysis of ABMs (which need not involve data at all), through laboratory experiments that record only behaviour to ethnographic collection and analysis of diverse data in rich experiments (like companion modelling exercises.) But it is important for progress that the field develops robust concepts that allow meaningful distinctions and does not get distracted into pointless arguments about labelling. Furthermore, we must consider the possible scientific implications of experimentation carried out at different points in the dimension space: For example, what are the relative strengths and limitations of experiments that are more or less controlled or more or less data diverse? Is there a “sweet spot” where the benefit of experiments is greatest to Agent-Based Modelling? If so, what is it and why?
The Philosophy of Experiment
The second challenge is the different beliefs (often associated with different disciplines) about the philosophical underpinnings of experiment such as what we might mean by a cause. In an economic experiment, for example, the objective may be to confirm a universal theory of decision making through displayed behaviour only. (It is decisions described by this theory which are presumed to cause the pattern of observed behaviour.) This will probably not allow the researcher to discover that their basic theory is wrong (people are adaptive not rational after all) or not universal (agents have diverse strategies), or that some respondents simply didn’t understand the experiment (deviations caused by these phenomena may be labelled noise relative to the theory being tested but in fact they are not.)
By contrast qualitative sociologists believe that subjective accounts (including accounts of participation in the experiment itself) can be made reliable and that they may offer direct accounts of certain kinds of cause: If I say I did something for a certain reason then it is at least possible that I actually did (and that the reason I did it is therefore its cause). It is no more likely that agreement will be reached on these matters in the context of experiments than it has been elsewhere. But Agent-Based Modelling should keep its reputation for open mindedness by seeing what happens when qualitative data is also collected and not just rejecting that approach out of hand as something that is “not done”. There is no need for Agent-Based Modelling blindly to follow the methodology of any one existing discipline in which experiments are conducted (and these disciplines often disagree vigorously on issues like payment and deception with no evidence on either side which should also make us cautious about their self-evident correctness.)
Finally, there is a further complication in understanding experiments using analogies with the physical sciences. In understanding the evolution of a river system, for example, one can control/intervene, one can base theories on testable micro mechanisms (like percolation) and one can observe. But there is no equivalent to asking the river what it intends (whether we can do this effectively in social science or not).[ii] It is not totally clear how different kinds of data collection like these might relate to each other in the social sciences, for example, data from subjective accounts, behavioural experiments (which may show different things from what respondents claim) and, for example, brain scans (which side step the social altogether.) This relationship between different kinds of data currently seems incompletely explored and conceptualised. (There is a tendency just to look at easy cases like surveys versus interviews.)
The Challenge of Experiments as Practical Research
This is an important area where the actual and potential users of experiments participating in the round table diverged. Potential users wanted clear guidance on the resources, skills and practices involved in doing experimental work (and see similar issues in the behavioural strategy literature, for example, Reypens and Levine 2018). At the most basic level, when does a researcher need to do an experiment (rather than a survey, interviews or observation), what are the resource requirements in terms of time, facilities and money (laboratory experiments are unusual in often needing specific funding to pay respondents rather than substituting the researcher working for free) what design decisions need to be made (paying subjects, online or offline, can subjects be deceived?), how should the data be analysed (how should an ABM be validated against experimental data?) and so on.[iii] (There are also pros and cons to specific bits of potentially supporting technology like Amazon Mechanical Turk, Qualtrics and Prolific, which have not yet been documented and systematically compared for the novice with a background in Agent-Based Modelling.) There is much discussion about these matters in the traditional literatures of social sciences that do experiments (see, for example, Kagel and Roth 1995, Levine and Parkinson 1994 and Zelditch 2014) but this has not been summarised and tuned specifically for the needs of Agent-Based Modellers (or published where they are likely to see it).
However, it should not be forgotten that not all research efforts need this integration within the same project, so thinking about the problems that really need it is critical. Nonetheless, triangulation is indeed necessary within research programmes. For instance, in subfields such as strategic management and organisational design, it is uncommon to see an ABM integrated with an experiment as part of the same project (though there are exceptions, such as Vuculescu 2017). Instead, ABMs are typically used to explore “what if” scenarios, build process theories and illuminate potential empirical studies. In this approach, knowledge is accumulated instead through the triangulation of different methodologies in different projects (see Burton and Obel 2018). Additionally, modelling and experimental efforts are usually led by different specialists – for example, there is a Theoretical Organisational Models Society whose focus is the development of standards for theoretical organisation science.
In a relatively new and small area, all we often have is some examples of good practice (or more contentiously bad practice) of which not everyone is even aware. A preliminary step is thus to see to what extent people know of good practice and are able to agree that it is good (and perhaps why it is good).
Finally, there was a slightly separate discussion about the perspectives of experimental participants themselves. It may be that a general problem with unreal activity is that you know it is unreal (which may lead to problems with ecological validity – Bornstein 1999.) On the other hand, building on the enrichment argument put forward by Edmonds (above), there is at least anecdotal observational evidence that richer and more realistic settings may cause people to get “caught up” and perhaps participate more as they would in reality. Nonetheless, there are practical steps we can take to learn more about these phenomena by augmenting experimental designs. For example we might conduct interviews (or even group discussions) before and after experiments. This could make the initial biases of participants explicit and allow them to self-evaluate retrospectively the extent to which they got engaged (or perhaps even over-engaged) during the game. The first such questionnaire could be available before attending the experiment, whilst another could be administered right after the game (and perhaps even a third a week later). In addition to practical design solutions, there are also relevant existing literatures that experimental researchers should probably draw on in this area, for example that on systemic design and the associated concept of worldviews. But it is fair to say that we do not yet fully understand the issues here but that they clearly matter to the value of experimental data for Agent-Based Modelling.[iv]
Design of Experiments
Something that came across strongly in the round table discussion as argued by existing users of experimental methods was the desirability of either designing experiments directly based on a specific ABM structure (rather than trying to use a stripped down – purely behavioural – experiment) or mixing real and simulated participants in richer experimental settings. In line with the enrichment argument put forward by Edmonds, nobody seemed to be using stripped down experiments to specify, calibrate or validate ABM elements piecemeal. In the examples provided by round table participants, experiments corresponding closely to the ABM (and mixing real and simulated participants) seemed particularly valuable in tackling subjects that existing theory had not yet really nailed down or where it was clear that very little of the data needed for a particular ABM was available. But there was no sense that there is a clearly defined set of research designs with associated purposes on which the potential user can draw. (The possible role of experiments in supporting policy was also mentioned but no conclusions were drawn.)
Extracting Rich Data from Experiments
Traditional experiments are time consuming to do, so they are frequently optimised to obtain the maximum power and discrimination between factors of interest. In such situations they will often limit their data collection to what is strictly necessary for testing their hypotheses. Furthermore, it seems to be a hangover from behaviourist psychology that one does not use self-reporting on the grounds that it might be biased or simply involve false reconstruction (rationalisation). From the point of view of building or assessing ABMs this approach involves a wasted opportunity. Due to the flexible nature of ABMs there is a need for as many empirical constraints upon modelling as possible. These constraints can come from theory, evidence or abstract principles (such as simplicity) but should not hinder the design of an ABM but rather act as a check on its outcomes. Game-like situations can provide rich data about what is happening, simultaneously capturing decisions on action, the position and state of players, global game outcomes/scores and what players say to each other (see, for example, Janssen et al. 2010, Lindahl et al. 2021). Often, in social science one might have a survey with one set of participants, interviews with others and longitudinal data from yet others – even if these, in fact, involve the same people, the data will usually not indicate this through consistent IDs. When collecting data from a game (and especially from online games) there is a possibility for collecting linked data with consistent IDs – including interviews – that allows for a whole new level of ABM development and checking.
Standards and Institutional Bootstrapping
This is also a wider problem in newer methods like Agent-Based Modelling. How can we foster agreement about what we are doing (which has to build on clear concepts) and institutionalise those agreements into standards for a field (particularly when there is academic competition and pressure to publish).[v] If certain journals will not publish experiments (or experiments done in certain ways) what can we do about that? JASSS was started because it was so hard to publish ABMs. It has certainly made that easier but is there a cost through less publication in other journals? See, for example, Squazzoni and Casnici (2013). Would it have been better for the rigour and wider acceptance of Agent-Based Modelling if we had met the standards of other fields rather than setting our own? This strategy, harder in the short term, may also have promoted communication and collaboration better in the long term. If reviewing is arbitrary (reviewers do not seem to have a common view of what makes an experiment legitimate) then can that situation be improved (and in particular how do we best go about that with limited resources?) To some extent, normal individualised academic work may achieve progress here (researchers make proposals, dispute and refine them and their resulting quality ensures at least some individualised adoption by other researchers) but there is often an observable gap in performance: Even though most modellers will endorse the value of data for modelling in principle most models are still non-empirical in practice (Angus and Hassani-Mahmooei 2015, Figure 9). The jury is still out on the best way to improve reviewer consistency, use the power of peer review to impose better standards (and thus resolve a collective action problem under academic competition[vi]) and so on but recognising and trying to address these issues is clearly important to the health of experimental methods in Agent-Based Modelling. Since running experiments in association with ABMs is already challenging, adding the problem of arbitrary reviewer standards makes the publication process even harder. This discourages scientists from following this path and therefore retards this kind of research generally. Again, here, useful resources (like the Psychological Science Accelerator, which facilitates greater experimental rigour by various means) were suggested in discussion as raw material for our own improvements to experiments in Agent-Based Modelling.
Another issue with newer methods such as Agent-Based Modelling is the path to legitimation before the wider scientific community. The need to integrate ABMs with experiments does not necessarily imply that the legitimation of the former is achieved by the latter. Experimental economists, for instance, may still argue that (in the investigation of behaviour and its implications for policy issues), experiments and data analysis alone suffice. They may rightly ask: What is the additional usefulness of an ABM? If an ABM always needs to be justified by an experiment and then validated by a statistical model of its output, then the method might not be essential at all. Orthodox economists skip the Agent-Based Modelling part: They build behavioural experiments, gather (rich) data, run econometric models and make predictions, without the need (at least as they see it) to build any computational representation. Of course, the usefulness of models lies in the premise that they may tell us something that experiments alone cannot (see Knudsen et al. 2019). But progress needs to be made in understanding (and perhaps reconciling) these divergent positions. The social simulation community therefore needs to be clearer about exactly what ABMs can contribute beyond the limitations of an experiment, especially when addressing audiences of non-modellers (Ballard et al. 2021). Not only is a model valuable when rigorously validated against data, but also whenever it makes sense of the data in ways that traditional methods cannot.
Where Now?
Researchers usually have more enthusiasm than they have time. In order to make things happen in an academic context it is not enough to have good ideas, people need to sign up and run with them. There are many things that stand a reasonable chance of improving the profile and practice of experiments in Agent-Based Modelling (regular sessions at SSC, systematic reviews, practical guidelines and evaluated case studies, discussion groups, books or journal special issues, training and funding applications that build networks and teams) but to a great extent, what happens will be decided by those who make it happen. The organisers of this round table (Nanda Wijermans and Edmund Chattoe-Brown) are very keen to support and coordinate further activity and this summary of discussions is the first step to promote that. We hope to hear from you.
References
Angus, Simon D. and Hassani-Mahmooei, Behrooz (2015) ‘“Anarchy” Reigns: A Quantitative Analysis of Agent-Based Modelling Publication Practices in JASSS, 2001-2012’, Journal of Artificial Societies and Social Simulation, 18(4), October, article 16, <http://jasss.soc.surrey.ac.uk/18/4/16.html>. doi:10.18564/jasss.2952
Ballard, Timothy, Palada, Hector, Griffin, Mark and Neal, Andrew (2021) ‘An Integrated Approach to Testing Dynamic, Multilevel Theory: Using Computational Models to Connect Theory, Model, and Data’, Organizational Research Methods, 24(2), April, pp. 251-284. doi: 10.1177/1094428119881209
Basole, Rahul C., Bodner, Douglas A. and Rouse, William B. (2013) ‘Healthcare Management Through Organizational Simulation’, Decision Support Systems, 55(2), May, pp. 552-563. doi:10.1016/j.dss.2012.10.012
Boero, Riccardo, Bravo, Giangiacomo, Castellani, Marco and Squazzoni, Flaminio (2010) ‘Why Bother with What Others Tell You? An Experimental Data-Driven Agent-Based Model’, Journal of Artificial Societies and Social Simulation, 13(3), June, article 6, <https://www.jasss.org/13/3/6.html>. doi:10.18564/jasss.1620
Bornstein, Brian H. (1999) ‘The Ecological Validity of Jury Simulations: Is the Jury Still Out?’ Law and Human Behavior, 23(1), February, pp. 75-91. doi:10.1023/A:1022326807441
Burton, Richard M. and Obel, Børge (2018) ‘The Science of Organizational Design: Fit Between Structure and Coordination’, Journal of Organization Design, 7(1), December, article 5. doi:10.1186/s41469-018-0029-2
Derbyshire, James (2020) ‘Answers to Questions on Uncertainty in Geography: Old Lessons and New Scenario Tools’, Environment and Planning A: Economy and Space, 52(4), June, pp. 710-727. doi:10.1177/0308518X19877885
Dinesen, Peter Thisted (2013) ‘Where You Come From or Where You Live? Examining the Cultural and Institutional Explanation of Generalized Trust Using Migration as a Natural Experiment’, European Sociological Review, 29(1), February, pp. 114-128. doi:10.1093/esr/jcr044
Doran, Jim (1998) ‘Simulating Collective Misbelief’, Journal of Artificial Societies and Social Simulation, 1(1), January, article 1, <https://www.jasss.org/1/1/3.html>.
Janssen, Marco A., Holahan, Robert, Lee, Allen and Ostrom, Elinor (2010) ‘Lab Experiments for the Study of Social-Ecological Systems’, Science, 328(5978), 30 April, pp. 613-617. doi:10.1126/science.1183532
Kagel, John H. and Roth, Alvin E. (eds.) (1995) The Handbook of Experimental Economics (Princeton, NJ: Princeton University Press).
Knudsen, Thorbjørn, Levinthal, Daniel A. and Puranam, Phanish (2019) ‘Editorial: A Model is a Model’, Strategy Science, 4(1), March, pp. 1-3. doi:10.1287/stsc.2019.0077
Levine, Gustav and Parkinson, Stanley (1994) Experimental Methods in Psychology (Hillsdale, NJ: Lawrence Erlbaum Associates).
Lindahl, Therese, Janssen, Marco A. and Schill, Caroline (2021) ‘Controlled Behavioural Experiments’, in Biggs, Reinette, de Vos, Alta, Preiser, Rika, Clements, Hayley, Maciejewski, Kristine and Schlüter, Maja (eds.) The Routledge Handbook of Research Methods for Social-Ecological Systems (London: Routledge), pp. 295-306. doi:10.4324/9781003021339-25
Ramanath, Ana Maria and Gilbert, Nigel (2004) ‘The Design of Participatory Agent-Based Social Simulations’, Journal of Artificial Societies and Social Simulation, 7(4), October, article 1, <https://www.jasss.org/7/4/1.html>.
Reypens, Charlotte and Levine, Sheen S. (2018) ‘Behavior in Behavioral Strategy: Capturing, Measuring, Analyzing’, in Behavioral Strategy in Perspective, Advances in Strategic Management Volume 39 (Bingley: Emerald Publishing), pp. 221-246. doi:10.1108/S0742-332220180000039016
Squazzoni, Flaminio and Casnici, Niccolò (2013) ‘Is Social Simulation a Social Science Outstation? A Bibliometric Analysis of the Impact of JASSS’, Journal of Artificial Societies and Social Simulation, 16(1), January, article 10, <http://jasss.soc.surrey.ac.uk/16/1/10.html>. doi:10.18564/jasss.2192
Taillandier, Patrick, Grignard, Arnaud, Marilleau, Nicolas, Philippon, Damien, Huynh, Quang-Nghi, Gaudou, Benoit and Drogoul, Alexis (2019) ‘Participatory Modeling and Simulation with the GAMA Platform’, Journal of Artificial Societies and Social Simulation, 22(2), March, article 3, <https://www.jasss.org/22/2/3.html>. doi:10.18564/jasss.3964
Tykhonov, Dmytro, Jonker, Catholijn, Meijer, Sebastiaan and Verwaart, Tim (2008) ‘Agent-Based Simulation of the Trust and Tracing Game for Supply Chains and Networks’, Journal of Artificial Societies and Social Simulation, 11(3), June, article 1, <https://www.jasss.org/11/3/1.html>.
Vuculescu, Oana (2017) ‘Searching Far Away from the Lamp-Post: An Agent-Based Model’, Strategic Organization, 15(2), May, pp. 242-263. doi:10.1177/1476127016669869
Zelditch, Morris Junior (2007) ‘Laboratory Experiments in Sociology’, in Webster, Murray Junior and Sell, Jane (eds.) Laboratory Experiments in the Social Sciences (New York, NY: Elsevier), pp. 183-197.
Notes
[i] This event was organised (and the resulting article was written) as part of “Towards Realistic Computational Models of Social Influence Dynamics” a project funded through ESRC (ES/S015159/1) by ORA Round 5 and involving Bruce Edmonds (PI) and Edmund Chattoe-Brown (CoI). More about SSC2021 (Social Simulation Conference 2021) can be found at https://ssc2021.uek.krakow.pl
[ii] This issue is actually very challenging for social science more generally. When considering interventions in social systems, knowing and acting might be so deeply intertwined (Derbyshire 2020) that interventions may modify the same behaviours that an experiment is aiming to understand.
[iii] In addition, experiments often require institutional ethics approval (but so do interviews, gaming activities and others sort of empirical research of course), something with which non-empirical Agent-Based Modellers may have little experience.
[iv] Chattoe-Brown had interesting personal experience of this. He took part in a simple team gaming exercise about running a computer firm. The team quickly worked out that the game assumed an infinite return to advertising (so you could have a computer magazine consisting entirely of adverts) independent of the actual quality of the product. They thus simultaneously performed very well in the game from the perspective of an external observer but remained deeply sceptical that this was a good lesson to impart about running an actual firm. But since the coordinators never asked the team members for their subjective view, they may have assumed that the simulation was also a success in its didactic mission.
[v] We should also not assume it is best to set our own standards from scratch. It may be valuable to attempt integration with existing approaches, like qualitative validity (https://conjointly.com/kb/qualitative-validity/) particularly when these are already attempting to be multidisciplinary and/or to bridge the gap between, for example, qualitative and quantitative data.
[vi] Although journals also face such a collective action problem at a different level. If they are too exacting relative to their status and existing practice, researchers will simply publish elsewhere.
Dino Carpentras, Edmund Chattoe-Brown, Bruce Edmonds, Cesar García-Diaz, Christian Kammler, Anna Pagani and Nanda Wijermans (2020) Where Now For Experiments In Agent-Based Modelling? Report of a Round Table as Part of SSC2021. Review of Artificial Societies and Social Simulation, 2nd Novermber 2021. https://rofasss.org/2021/11/02/round-table-ssc2021-experiments/
The initiative presented below arose from a Lorentz Center workshop on Integrating Qualitative and Quantitative Evidence using Social Simulation (8-12 April 2019, Leiden, the Netherlands). At the beginning of this workshop, the attenders divided themselves into teams aiming to work on specific challenges within the broad domain of the workshop topic. Our team took up the challenge of looking at “Rigour, Transparency, and Reuse”. The aim that emerged from our initial discussions was to create a framework for augmenting rigour and transparency (RAT) of data use in ABM when both designing, analysing and publishing such models.
One element of the framework that the group worked on was a roadmap of the modelling process in ABM, with particular reference to the use of different kinds of data. This roadmap was used to generate the second element of the framework: A protocol consisting of a set of questions, which, if answered by the modeller, would ensure that the published model was as rigorous and transparent in terms of data use, as it needs to be in order for the reader to understand and reproduce it.
The group (which had diverse modelling approaches and spanned a number of disciplines) recognised the challenges of this approach and much of the week was spent examining cases and defining terms so that the approach did not assume one particular kind of theory, one particular aim of modelling, and so on. To this end, we intend that the framework should be thoroughly tested against real research to ensure its general applicability and ease of use.
The team was also very keen not to “reinvent the wheel”, but to try develop the RAT approach (in connection with data use) to augment and “join up” existing protocols or documentation standards for specific parts of the modelling process. For example, the ODD protocol (Grimm et al. 2010) and its variants are generally accepted as the established way of documenting ABM but do not request rigorous documentation/justification of the data used for the modelling process.
The plan to move forward with the development of the framework is organised around three journal articles and associated dissemination activities:
A literature review of best (data use) documentation and practice in other disciplines and research methods (e.g. PRISMA – Preferred Reporting Items for Systematic Reviews and Meta-Analyses)
A literature review of available documentation tools in ABM (e.g. ODD and its variants, DOE, the “Info” pane of NetLogo, EABSS)
An initial statement of the goals of RAT, the roadmap, the protocol and the process of testing these resources for usability and effectiveness
A presentation, poster, and round table at SSC 2019 (Mainz)
We would appreciate suggestions for items that should be included in the literature reviews, “beta testers” and critical readers for the roadmap and protocol (from as many disciplines and modelling approaches as possible), reactions (whether positive or negative) to the initiative itself (including joining it!) and participation in the various activities we plan at Mainz. If you are interested in any of these roles, please email Melania Borit (melania.borit@uit.no).
Acknowledgements
Chattoe-Brown’s contribution to this research is funded by the project “Towards Realistic Computational Models Of Social Influence Dynamics” (ES/S015159/1) funded by ESRC via ORA Round 5 (PI: Professor Bruce Edmonds, Centre for Policy Modelling, Manchester Metropolitan University: https://gtr.ukri.org/projects?ref=ES%2FS015159%2F1).
References
Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J. and Railsback, S. F. (2010) ‘The ODD Protocol: A Review and First Update’, Ecological Modelling, 221(23):2760–2768. doi:10.1016/j.ecolmodel.2010.08.019
Achter, S., Borit, M., Chattoe-Brown, E., Palaretti, C. and Siebers, P.-O.(2019) Cherchez Le RAT: A Proposed Plan for Augmenting Rigour and Transparency of Data Use in ABM. Review of Artificial Societies and Social Simulation, 4th June 2019. https://rofasss.org/2019/06/04/rat/
Review of Artificial Societies and Social Simulation 