Tag Archives: generality

Content

Delusional Generality – how models can give a false impression of their applicability even when they lack any empirical foundation

Leave a comment

By Bruce Edmonds1, Dino Carpentras2, Nick Roxburgh3, Edmund Chattoe-Brown4 and Gary Polhill3

  1. Centre for Policy Modelling, Manchester Metropolitan University
  2. Computational Social Science, ETH Zurich
  3. James Hutton Institute, Aberdeen
  4. University of Leicester

“Hamlet: Do you see yonder cloud that’s almost in shape of a camel?
Polonius: By the mass, and ‘tis like a camel, indeed.
Hamlet: Methinks it is like a weasel.
Polonius: It is backed like a weasel.
Hamlet: Or like a whale?
Polonius: Very like a whale.”

Models and Generality

The essence of a model is that it represents – if it is not a model of something it is not a model at all (Zeigler 1976, Wartofsky 1979). A random bit of code or set of equations is not a model. The point of a model is that one can use the model to infer or understand some aspects about what it represents. However, models can represent a variety of kinds of things in a variety of ways (Edmonds & al. 2019) – it can represent ideas, correspond to data, or aspects of other models and it can represent each of these in either a vague or precise manner. To completely understand a model – its construction, properties and working – one needs to understand how it does this mapping. This piece focuses attention on this mapping, rather than the internal construction of models.

What a model reliably represents may be a single observed situation, but it might satisfactorily represent more than one such situation. The range of situations that the model satisfactorily represents is called the “scope” of the model (what is “satisfactory” depending on the purpose for which the model is being used). The more extensive the scope, the more “general” we say the model is. A model that only represents one case has no generality at all and may be more in the nature of a description.

There is a hunger for general accounts of social phenomena (let us call these ‘theories’). However, this hunger is often frustrated by the sheer complexity and ‘messiness’ involved in such phenomena. If every situation we observe is essentially different, then no such theory is possible. However, we hope that this is not the case for the social world and, indeed, informal observation suggests that there is, at least some, commonality between situations – in other words, that some kind of reliable generalisation about social phenomena might be achievable, however modest (Merton 1968). This piece looks at two kinds of applicability – analogical applicability and empirical applicability – and critiques those that conflate them. Although the expertise of the authors is in the agent-based modelling of social phenomena, and so we restrict our discussion to this, we strongly suspect that our arguments are true for many kinds of modelling across a range of domains.

In the next sections we contrast two uses for models: as analogies (ways of thinking about observed systems) and those that intend to represent empirical data in a more precise way. There are, of course, other uses of model such as that of exploring theory which have nothing to do with anything observed.

Models used as analogies

Analogical applicability comes from the flexibility of the human mind in interpreting accounts in terms of the different situations. When we encounter a new situation, the account is mapped onto it – the account being used as an analogy for understanding this situation. Such accounts are typically in the form of a narrative, but a model can also be used as an analogy (which is the case we are concerned with here). The flexibility with which this mapping can be constructed means that such an account can be related to a wide range of phenomena. Such analogical mapping can lead to an impression that the account has a wide range of applicability. Analogies are a powerful tool for thinking since it may give us some insights into otherwise novel situations. There are arguments that analogical thinking is a fundamental aspect of human thought (Hofstadter 1995) and language (Lakoff 2008). We can construct and use analogical mappings so effortlessly that they seem natural to us. The key thing about analogical thinking is that the mapping from the analogy to the situation to which it is applied is re-invented each time – there is no fixed relationship between the analogy and what it might be applied to. We are so good at doing this that we may not be aware of how different the constructed mapping is each time. However, its flexibility comes at a cost, namely that because there is no well-defined relationship with what it applies to, the mapping tends to be more intuitive than precise. An analogy can give insights but analogical reasoning suggests rather than establishes anything reliably and you cannot empirically test it (since analogical mappings can be adjusted to avoid falsification). Such “ways of thinking” might be helpful, but equally might be misleading [note ‎1].

Just because the content of an analogy might be expressed formally does not change any of this (Edmonds 2018), in fact formally expressed analogies might give the impression of being applicable, but often are only related to anything observed via ideas – the model relates to some ideas, and the ideas relate to reality (Edmonds 2000). Using models as analogies is a valid use of models but this is not an empirically reliable one (Edmonds et al. 2019). Arnold (2013) makes a powerful argument that many of the more abstract simulation models are of this variety and simply not relatable to empirically observed cases and data at all – although these give the illusion of wide applicability, that applicability is not empirical. In physics the ways of thinking about atomic or subatomic entities have changed over time whilst the mathematically-expressed, empirically-relevant models have not (Hartman 1997). Although Thompson (2022) concentrates on mathematically formulated models, she also distinguishes between well-validated empirical models and those that just encapsulate the expertise/opinion of the modeller. She gives some detailed examples of where the latter kind had disproportionate influence, beyond that of other expertise, just because it was in the form of a model (e.g. the economic impact of climate change).

An example of an analogical model is described in Axelrod (1984) – a formalised tournament where algorithmically-expressed strategies are pitted against each other, playing the iterated prisoner’s dilemma game. It is shown how the ‘tit for tat’ strategy can survive against many other mixes of strategies (static or evolving).  In the book, the purpose of the model is to suggest a new way of thinking about the evolution of cooperation. The book claims the idea ‘explains’ many observed phenomena, but this in an analogical manner – no precise relationship with any observed measurements is described. There is no validation of the model here or in the more academic paper that described these results (Axelrod & Hamilton 1981).

Of course, researchers do not usually call their models “analogies” or “analogical” explicitly but tend to use other phrasings that imply a greater importance. An exception is Epstein (2008) where it is explicitly listed as one of the 15 modelling purposes, other than prediction, that he discusses. Here he says such models are “…more than beautiful testaments to the unifying power of models: they are headlights in dark unexplored territory.” (ibid.) thus suggesting their use in thinking about phenomena where we do not already have reliable empirical models. Anything that helps us think about such phenomena could be useful, but that does not mean they are at all reliable. As Herbert Simon said: “Metaphor and analogy can be helpful, or they can be misleading. ” (Simon 1968, p. 467).

Another purpose listed in Epstein (2008) is to “Illuminate core dynamics”. After raising the old chestnut that “All models are wrong”, he goes on to justify them on the grounds that “…they capture qualitative behaviors of overarching interest”. This is fine if the models are, in fact, known to be useful as more than vague analogies [Note 2] – that they do, in some sense, approximate observed phenomena – but this is not the case with novel models that have not been empirically tested. This phrase is more insidious, because it implies that the dynamics that have been illuminated by the model are “core” – some kind of approximation of what is important about the phenomena, allowing for future elaborations to refine the representation. This implies a process where an initially rough idea is iteratively improved. However, this is premature because we do not know if what has been abstracted away in the abstract model was essential to the dynamics of the target phenomena or not without empirical testing – this is just assumed or asserted based on the intuitions of the modeller.

This idea of the “core dynamics” leads to some paradoxical situations – where a set of competing models are all deemed to be core. Indeed, the literature has shown how the same phenomenon can be modelled in many contrasting ways. For instance, political polarisation has been modelled through models with mechanisms for repulsion, bounded confidence, reinforcement, or even just random fluctuations, to name a few (Flache et al., 2017; Banisch & Olbrich 2019; Carpentras et al. 2022). However, it is likely that only a few of them contribute substantially to the political polarisation we observe in the real world, and so that all the others are not a real “core dynamic” but until we have more empirical work we do not know which are core and which not.

A related problem with analogical models is that, even when relying on parsimony principles [Note 3], it is not possible to decide which model is better. This aspect, combined with the constant production of new models, can makes the relevant literature increasingly difficult to navigate as models proliferate without any empirical selection, especially for researchers new to ABM. Furthermore, most analogical models define their object of study in an imprecise manner so that it is hard to evaluate whether they are even intended to capture element of any particular observed situation. For example, opinion dynamics models rarely define the type of interaction they represent (e.g. in person vs online) or even what an opinion is. This has led to cases where even knowledge of facts has been studied as “opinions” (e.g. Chacoma & Zanette, 2015).

In summary, analogical models can be a useful tool to start thinking about complex phenomena. However, the danger with them is that they give an impression of progress but result in more confusion than clarity, possibly slowing down scientific progress. Once one has some possible insights, one needs to confront these with empirical data to determine which are worth further investigation.

Models that relate directly to empirical data

An empirical model, in contrast, has a well-defined way of mapping to the phenomena it represents. For example, the variables of the gas laws (volume, temperature and pressure) are measured using standard methods developed over a long period of time, one does not invent a new way of doing this each time the laws are applied. In this case, the ways of measuring these properties have developed alongside the mathematical models of the laws so that these work reliably under broad (and well known) conditions and cannot be adjusted at the whim of a modeller. Empirical generality comes from when a model applies reliably to many different situations – in the case of the gas laws, to a wide range of materials in gaseous form to a high degree of accuracy.

Empirical models can be used for different purposes, including: prediction, explanation and description (Edmonds et al. 2019). Each of these uses how the model is mapped to empirical data in different ways, to reflect these purposes. With a descriptive model the mapping is one-way from empirical data to the model to justify the different parts. In a predictive model, the initial model setup is determined from known data and the model is then run to get its results. These results are then mapped back to what we might expect as a prediction, which can be later compared to empirically measured values to check the model’s validity. An explanatory model supports a complex explanation of some known outcomes in terms of a set of processes, structures and parameter values. When it is shown that the outcomes of such a model sufficiently match those from the observed data – the model represents a complex chain of causation that would result in that data in terms of the processes, structures and parameter values it comprised. It thus supports an explanation in terms of the model and its input of what was observed. In each of these three cases the mapping from empirical data to the model happens in a different order and maybe in a different direction, however they all depend upon the mapping being well defined.

Cartwright (1983), studying how physics works, distinguished between explanatory and phenomenological laws – the former explains but does not necessary relate exactly to empirical data (such as when we fit a line to data using regression), whilst the latter fits the data but does not necessarily explain (like the gas laws). Thus the jobs of theoretical explanation and empirical prediction are done by different models or theories (often calling the explanatory version “theory” and the empirical versions “models”). However, in physics the relationship between the two is, itself, examined so that the “bridging laws” between them are well understood, especially in formal terms. In this case, we attribute reliable empirical meaning to the explanatory theories to the extent that the connection to the data is precise, even though it is done via the intermediary of an “phenomenological” model because both mappings (explanatory↔phenomenological and phenomenological↔empirical data) are precise and well established. The point is that the total mapping from model or theory to empirical data is not subject to interpretation or post-hoc adjustment to improve its fit.

ABMs are often quite complicated and require many parameters or other initialising input to be specified before they can be run. If some of these are not empirically determinable (even in principle) then these might be guessed at using a process of “calibration”, that is searching the space of possible initialisations for some values for which some measured outcomes of the results match other empirical data. If the model has been separately shown to be empirically reliable then one could do such a calibration to suggest what these input values might have been. Such a process might establish that the model captures a possible explanation of the fitted outcomes (in terms of the model plus those backward-inferred input values), but this is not a very strong relationship, since many models are very flexible and so could fit a wide range of possible outcomes. The reliability of such a suggested explanation, supported by the model, is only relative to (a) the empirical reliability of any theory or other assumptions the model is built upon (b) how flexibly the model outcomes can be adjusted to fit the target data and (c) how precisely the choice of outcome measures and fit are. Thus, calibration does not provide strong evidence of the empirical adequacy of an ABM and any explanation supported by such a procedure is only relative to the ‘wiggle room’ afforded by free parameters and unknown input data as well as any assumptions used in the making of the model. However, empirical calibration is better than none and may empirically fix the context in which theoretical exploration occurs – showing that the model is, at least, potentially applicable to the case being considered [Note 4].

An example of a model that is strongly grounded in empirical data is the “538” model of the US electoral college for presidential elections (Silver 2012). This is not an ABM but more like a micro-simulation. It aggregates the uncertainty from polling data to make probabilistic predictions about what this means for the outcomes. The structure of the model comes directly from the rules of the electoral college, the inputs are directly derived from the polling data and it makes predictions about the results that can be independently checked. It does a very specific, but useful job, in translating the uncertainty of the polling data into the uncertainty about the outcome.

Why this matters

If people did not confuse the analogical and empirical cases, there would not be a problem. However, researchers seem to suffer from a variety of “Kuhnian Spectacles” (Kuhn 1962) – namely that because they view their target systems through an analogical model, they tend to think that this is how that system actually is – i.e. that the model has not just analogical but also empirical applicability. This is understandable, we use many layers of analogy to navigate our world and in many every-day cases it is practical to conflate our models with the reality we deal with (when they are very reliable). However, people who claim to be scientists are under an obligation to be more cautious and precise than this, since others might wish to rely upon our theories and models (this is, after all, why they support us in our privileged position). However, such caution is not always followed. There are cases where modellers declare their enterprise a success even after a long period without any empirical backing, making a variety of excuses instead of coming clean about this lack (Arnold 2015).

Another fundamental aspect is that agent-based models can be very interdisciplinary and, because of that, they can be used also by researchers in different fields. However, many fields do not consider models as simple analogies, especially when they provide precise mathematical relationship among variables. This can easily result in confusions where the analogical applicability of ABMs is interpreted as empirical in another field.

Of course, we may be hopeful that, sometime in the future, our vague or abstract analogical model maybe developed into something with proven empirical abilities, but we should not suggest such empirical abilities until these have been established. Furthermore, we should be particularly careful to ensure that non-modellers understand that this possibility is only a hope and not imply anything otherwise (e.g. imply that it is likely to have empirical validity). However, we suspect that in many cases this confusion goes beyond optimistic anticipation and that some modellers conflate analogical with empirical applicability, assuming that their model is basically right just because it seems that way to them. This is what we call “delusional generality” – that a researcher is under the impression that their model has a wide applicability (or potentially wide applicability) due to the attractiveness of the analogy it presents. In other words, unaware of the unconscious process of re-inventing the mapping to each target system, they imagine (without further justification) that it has some reliable empirical (or potentially empirical) generality at its core [Note 5].

Such confusion can have severe real-world consequences if a model with only analogical validity is assumed to also have some empirical reliability. Thompson (2022) discusses how abstract economic models of the cost of future climate change did affect the debate about the need for prevention and mitigation, even though they had no empirical validity. However, agent-based modellers have also made the same mistake, with a slew of completely unvalidated models about COVID affecting public debate about policy (Squazzoni et al 2021).

Conclusion

All of the above discussion raises the question of how we might achieve reliable models with even a moderate level of empirical generality in the social sciences. This is a tricky question of scientific strategy, which we are not going to answer here [Note 6]. However, we question whether the approach of making “heroic” jumps from phenomena to abstract non-empirical models on the sole basis of its plausibility to its authors will be a productive route when the target is complex phenomena, such as socio-cognitive systems (Dignum, Edmonds and Carpentras 2022). Certainly, that route has not yet been empirically demonstrated.

Whatever the best strategy is, there is a lot of theoretical modelling in the field of social simulation that assumes or implies that it is the precursor for empirical applicability and not a lot of critique about the extent of empirical success achieved. The assumption seems to be that abstract theory is the way to make progress understanding social phenomena but, as we argue here, this is largely wishful thinking – the hope that such models will turn out to have empirical generality being a delusion.  Furthermore, this approach has substantive deleterious effects in terms of encouraging an explosion of analogical models without any process of selection (Edmonds 2010). It seems that the ‘famine’ of theory about social phenomena with any significant level of generality is so severe, that many seem to give credence to models they might otherwise reject – constructing their understanding using models built on sand.

Notes

1. There is some debate about the extent to which analogical reasoning works, what kind of insights it results in and under what circumstances (Hofstede 1995). However, all we need for our purposes is that: (a) it does not reliably produce knowledge, (b) the human mind is exceptionally good at ‘fitting’ analogies to new situations (adjusting the mapping to make it ‘work’ somehow) and (c) due to this ability analogies can be far more convincing that the analogical reasoning warrants.

2. In pattern-oriented modelling (Grimm & al 2005) models are related to empirical evidence in a qualitative (pattern-based) manner, for example to some properties of a distribution of numeric outcomes. In this kind of modelling, a precise numerical correspondence is replaced by a set of qualitative correspondences in many different dimensions. In this the empirical relevance of a model is established on the basis that it is too hard to simultaneously fit a model to evidence in this way, thus ruling that out as a source of its correspondence with that evidence.

3. So-called “parsimony principles” are a very unreliable manner of evaluating competing theories on grounds other than convenience or that of using limited data to justify the values of parameters (Edmonds 2007).

4. In many models a vague argument for its plausibility is often all that is described to show that it is applicable to the cases being discussed. At least calibration demonstrates its empirical applicability, rather than simply assuming it.

5. We are applying the principle of charity here, assuming that such conflations are innocent and not deliberate. However, there is increasing pressure from funding agencies to demonstrate ‘real life relevance’ so some of these apparent confusions might be more like ‘spin’ – trying to give an impression of empirical relevance even when this is merely an aspiration, in order to suggest that their model has more significant than they have reliably established.

6. This has been discussed elsewhere, e.g. (Moss & Edmonds 2005).

Acknowledgements

Thanks to all those we have discussed these issues with, including Scott Moss (who was talking about these kinds of issue more than 30 years ago), Eckhart Arnold (who made many useful comments and whose careful examination of the lack of empirical success of some families of model demonstrates our mostly abstract arguments), Sven Banisch and other members of the ESSA special interest group on “Strongly Empirical Modelling”.

References

Arnold, E. (2013). Simulation models of the evolution of cooperation as proofs of logical possibilities. How useful are they? Ethics & Politics, XV(2), pp. 101-138. https://philpapers.org/archive/ARNSMO.pdf

Arnold, E. (2015) How Models Fail – A Critical Look at the History of Computer Simulations of the Evolution of Cooperation. In Misselhorn, C. (Ed.): Collective Agency and Cooperation in Natural and Artificial Systems. Explanation, Implementation and Simulation, Philosophical Studies Series, Springer, pp. 261-279. https://eckhartarnold.de/papers/2015_How_Models_Fail

Axelrod, R. (1984) The Evolution of Cooperation, Basic Books.

Axelrod, R.  & Hamilton, W.D. (1981) The evolution of cooperation. Science, 211, 1390-1396. https://www.science.org/doi/abs/10.1126/science.7466396

Banisch, S., & Olbrich, E. (2019). Opinion polarization by learning from social feedback. The Journal of Mathematical Sociology, 43(2), 76-103. https://doi.org/10.1080/0022250X.2018.1517761

Carpentras, D., Maher, P. J., O’Reilly, C., & Quayle, M. (2022). Deriving An Opinion Dynamics Model From Experimental Data. Journal of Artificial Societies & Social Simulation, 25(4).http://doi.org/10.18564/jasss.4947

Cartwright, N. (1983) How the Laws of Physics Lie. Oxford University Press.

Chacoma, A. & Zanette, D. H. (2015). Opinion formation by social influence: From experiments to modelling. PloS ONE, 10(10), e0140406.https://doi.org/10.1371/journal.pone.0140406

Dignum, F., Edmonds, B. and Carpentras, D. (2022) Socio-Cognitive Systems – A Position Statement. Review of Artificial Societies and Social Simulation, 2nd Apr 2022. https://rofasss.org/2022/04/02/scs

Edmonds, B. (2000). The Use of Models – making MABS actually work. In. S. Moss and P. Davidsson. Multi Agent Based Simulation. Berlin, Springer-Verlag. 1979: 15-32. http://doi.org/10.1007/3-540-44561-7_2

Edmonds, B. (2007) Simplicity is Not Truth-Indicative. In Gershenson, C.et al. (eds.) Philosophy and Complexity. World Scientific, pp. 65-80.

Edmonds, B. (2010) Bootstrapping Knowledge About Social Phenomena Using Simulation Models. Journal of Artificial Societies and Social Simulation, 13(1), 8. http://doi.org/10.18564/jasss.1523

Edmonds, B. (2018) The “formalist fallacy”. Review of Artificial Societies and Social Simulation, 11th June 2018. https://rofasss.org/2018/07/20/be/

Edmonds, B., le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root H. & Squazzoni. F. (2019) Different Modelling Purposes. Journal of Artificial Societies and Social Simulation, 22(3):6. http://doi.org/10.18564/jasss.3993

Epstein, J. M. (2008). Why Model?. Journal of Artificial Societies and Social Simulation, 11(4),12. https://www.jasss.org/11/4/12.html

Flache, A., Mäs, M., Feliciani, T., Chattoe-Brown, E., Deffuant, G., Huet, S. & Lorenz, J. (2017). Models of social influence: Towards the next frontiers. Journal of Artificial Societies and Social Simulation, 20(4), 2. http://doi.org/10.18564/jasss.4298

Grimm, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W.M., Railsback, S.F., et al. (2005). Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science, 310 (5750), 987–991. https://www.jstor.org/stable/3842807

Hartman, S. (1997) Modelling and the Aims of Science. 20th International Wittgenstein Symposium, Kirchberg am Weshsel.

Hofstadter, D. (1995) Fluid Concepts and Creative Analogies. Basic Books.

Kuhn, T. S. (1962). The structure of scientific revolutions. University of Chicago Press.

Lakoff, G. (2008). Women, fire, and dangerous things: What categories reveal about the mind. University of Chicago Press.

Merton, R.K. (1968). On the Sociological Theories of the Middle Range. In Classical Sociological Theory, Calhoun, C., Gerteis, J., Moody, J., Pfaff, S. and Virk, I. (Eds), Blackwell, pp. 449–459.

Meyer, R. & Edmonds, B. (2023). The Importance of Dynamic Networks Within a Model of Politics. In: Squazzoni, F. (eds) Advances in Social Simulation. ESSA 2022. Springer Proceedings in Complexity. Springer. (Earlier, open access, version at: https://cfpm.org/discussionpapers/292)

Moss, S. and Edmonds, B. (2005). Towards Good Social Science. Journal of Artificial Societies and Social Simulation, 8(4), 13. https://www.jasss.org/8/4/13.html

Squazzoni, F. et al. (2020) ‘Computational Models That Matter During a Global Pandemic Outbreak: A Call to Action’ Journal of Artificial Societies and Social Simulation 23(2):10. http://doi.org/10.18564/jasss.4298

Silver, N, (2012) The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t. Penguin.

Simon, H. A. (1962). The architecture of complexity. Proceedings of the American philosophical society, 106(6), 467-482.https://www.jstor.org/stable/985254

Thompson, E. (2022). Escape from Model Land: How mathematical models can lead us astray and what we can do about it. Basic Books.

Wartofsky, M. W. (1979). The model muddle: Proposals for an immodest realism. In Models (pp. 1-11). Springer, Dordrecht.

Zeigler, B. P. (1976). Theory of Modeling and Simulation. Wiley Interscience, New York.

Edmonds, B., Carpentras, D., Roxburgh, N., Chattoe-Brown, E. and Polhill, G. (2024) Delusional Generality – how models can give a false impression of their applicability even when they lack any empirical foundation. Review of Artificial Societies and Social Simulation, 7 May 2024. https://rofasss.org/2024/05/06/delusional-generality

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

The Poverty of Suggestivism – the dangers of “suggests that” modelling

2 Comments

By Bruce Edmonds

Vagueness and refutation

A model[1] is basically composed of two parts (Zeigler 1976, Wartofsky 1979):

  1. A set of entities (such as mathematical equations, logical rules, computer code etc.) which can be used to make some inferences as to the consequences of that set (usually in conjunction with some data and parameter values)
  2. A mapping from this set to what it aims to represent – what the bits mean

Whilst a lot of attention has been paid to the internal rigour of the set of entities and the inferences that are made from them (1), the mapping to what that represents (2) has often been left as implicit or incompletely described – sometimes only indicated by the labels given to its parts. The result is a model that vaguely relates to its target, suggesting its properties analogically. There is not a well-defined way that the model is to be applied to anything observed, but a new map is invented each time it is used to think about a particular case. I call this way of modelling “Suggestivism”, because the model “suggests” things about what is being modelled.

This is partly a recapitulation of Popper’s critique of vague theories in his book “The Poverty of Historicism” (1957). He characterised such theories as “irrefutable”, because whatever the facts, these theories could be made to fit them. Irrefutability is an indicator of a lack of precise mapping to reality – such vagueness makes refutation very hard. However, it is only an indicator; there may be other reasons than vagueness for it not being possible to test a theory – it is their disconnection from well-defined empirical reference that is the issue here.

Some might go as far as suggesting that any model or theory that is not refutable is “unscientific”, but this goes too far, implying a very restricted definition of what ‘science’ is. We need analogies to think about what we are doing and to gain insight into what we are studying, e.g. (Hartman 1997) – for humans they are unavoidable, ‘baked’ into the way language works (Lakoff 1987). A model might make a set of ideas clear and help map out the consequences of a set of assumptions/structures/processes. Many of these suggestivist models relate to a set of ideas and it is the ideas that relate to what is observed (albeit informally) (Edmonds 2001). However, such models do not capture anything reliable about what they refer to, and in that sense are not part of the set of the established statements and theories that is at the core of science  (Arnold 2014).

The dangers of suggestivist modelling

As above, there are valid uses of abstract or theoretical modelling where this is explicitly acknowledged and where no conclusions about observed phenomena are made. So what are the dangers of suggestivist modelling – why am I making such a fuss about it?

Firstly, that people often seem to confuse a model as an analogy – a way of thinking about stuff – and a model that tells us reliably about what we are studying. Thus they give undue weight to the analyses of abstract models that are, in fact, just thought experiments. Making models is a very intimate way of theorising – one spends an extended period of time interacting with one’s model: developing, checking, analysing etc. The result is a particularly strong version of “Kuhnian Spectacles” (Kuhn 1962) causing us to see the world though our model for weeks after. Under this strong influence it is natural to confuse what we can reliably infer about the world and how we are currently perceiving/thinking about it. Good scientists should then pause and wait for this effect to wear off so that they can effectively critique what they have done, its limitations and what its implications are. However, often in the rush to get their work out, modellers often do not do this, resulting in a sloppy set of suggestive interpretations of their modelling.

Secondly, empirical modelling is hard. It is far easier (and, frankly, more fun) to play with non-empirical models. A scientific culture that treats suggestivist modelling as substantial progress and significantly rewards modellers that do it, will effectively divert a lot of modelling effort in this direction. Chattoe-Brown (2018) displayed evidence of this in his survey of opinion dynamics models – abstract, suggestivist modelling got far more reward (in terms of citations) than those that tried to relate their model to empirical data in a direct manner. Abstract modelling has a role in science, but if it is easier and more rewarding then the field will become unbalanced. It may give the impression of progress but not deliver on this impression. In a more mature science, researchers working on measurement methods (steps from observation to models) and collecting good data are as important as the theorists (Moss 1998).

Thirdly, it is hard to judge suggestivist models. Given their connection to the modelling target is vague there cannot be any decisive test of its success. Good modellers should declare the exact purpose of their model, e.g. that is analogical or merely exploring the consequences of theory (Edmonds et al. 2019), but then accept the consequences of this choice – namely, that it excludes  making conclusions about the observed world. If it is for a theoretical exploration then the comprehensiveness of the exploration, the scope of the exploration and the applicability of the model can be judged, but if the model is analogical or illustrative then this is harder. Whilst one model may suggest X, another may suggest the opposite. It is quite easy to fix a model to get the outcomes one wants. Clearly, if a model makes startling suggestions – illustrating totally new ideas or making a counter-example to widely held assumptions – then this helps science by widening the pool of theories or hypotheses that are considered. However most suggestivist modelling does not do this.

Fourthly, their sheer flexibility of as to application causes problems – if one works hard enough one can invent mappings to a wide range of cases, the limits are only those of our imagination. In effect, having a vague mapping from model to what it models adds in huge flexibility in a similar way to having a large number of free (non-empirical) parameters. This flexibility gives an impression of generality, and many desire simple and general models for complex phenomena. However, this is illusory because a different mapping is needed for each case, to make it apply. Given the above (1)+(2) definition of a model this means that, in fact, it is a different model for each case – what a model refers to, is part of the model. The same flexibility makes such models impossible to refute, since one can just adjust the mapping to save them. The apparent generality and lack of refutation means that such models hang around in the literature, due to their surface attractiveness.

Finally, these kinds of model are hugely influential beyond the community of modellers to the wider public including policy actors. Narratives that start in abstract models make their way out and can be very influential (Vranckx 1999). Despite the lack of rigorous mapping from model to reality, suggestivist models look impressive, look scientific. For example, very abstract models from the Neo-Classical ‘Chicago School’ of economists supported narratives about the optimal efficiency of markets, leading to a reluctance to regulate them (Krugman 2009). A lack of regulation seemed to be one of the factors behind the 2007/8 economic crash (Baily et al 2008). Modellers may understand that other modellers get over-enthusiastic and over-interpret their models, but others may not. It is the duty of modellers to give an accurate impression of the reliability of any modelling results and not to over-hype them.

How to recognise a suggestivist model

It can be hard to detangle how empirically vague a model is, because many descriptions about modelling work do not focus on making the mapping to what it represents precise. The reasons for this are various, for example: the modeller might be conflating reality and what is in the model in their minds, the researcher is new to modelling and has not really decided what the purpose of their model is, the modeller might be over-keen to establish the importance of their work and so is hyping the motivation and conclusions, they might simply not got around to thinking enough about the relationship between their model and what it might represent, or they might not have bothered to make the relationship explicit in their description. Whatever the reason the reader of any description of such work is often left with an archaeological problem: trying to unearth what the relationship might be, based on indirect clues only. The only way to know for certain is to take a case one knows about and try and apply the model to it, but this is a time consuming process and relies upon having a case with suitable data available. However, there are some indicators, albeit fallible ones, including the following.

  • A relatively simple model is interpreted as explaining a wide range of observed, complex phenomena
  • No data from an observed case study is compared to data from the model (often no data is brought in at all, merely abstract observations) – despite this, conclusions about some observed phenomena are made
  • The purpose of the model is not explicitly declared
  • The language of the paper seems to conflate talking about the model with what is being modelled
  • In the paper there are sudden abstraction ‘jumps’ between the motivation and the description of the model and back again to the interpretation of the results in terms of that motivation. The abstraction jumps involved are large and justified by some a priori theory or modelling precedents rather than evidence.

How to avoid suggestivist modelling

How to avoid the dangers of suggestivist modelling should be clear from the above discussion, but I will make them explicit here.

  • Be clear about the model purpose – that is does the model aim to achieve, which indicates how it should be judged by others (Edmonds et al 2019)
  • Do not make any conclusions about the real world if you have not related the model to any data
  • Do not make any policy conclusions – things that might affect other people’s lives – without at least some independent validation of the model outcomes
  • Document how a model relates (or should relate) to data, the nature of that data and maybe even the process whereby that data should be obtained (Achter et al 2019)
  • Be explicit as possible about what kinds of phenomena the model applies to – the limits of its scope
  • Keep the language about the model and what is being modelled distinct – for any statement it should be clear whether it is talking about the model or what it models (Edmonds 2020)
  • Highlight any bold assumptions in the specification of the model or describe what empirical foundation there is for them – be honest about these

Conclusion

Models can serve many different purposes (Epstein 2008). This is fine as long as the purpose of models are always made clear, and model results are not interpreted further than their established purpose allows. Research which gives the impression that analogical, illustrative or theoretical modelling can tell us anything reliable about observed complex phenomena is not only sloppy science, but can have a deleterious impact – giving an impression of progress whilst diverting attention from empirically reliable work. Like a bad investment: if it looks too good and too easy to be true, it probably isn’t.

Notes

[1] We often use the word “model” in a lazy way to indicate (1) rather than (1)+(2) in this definition, but a set of entities without any meaning or mapping to anything else is not a model, as it does not represent anything. For example, a random set of equations or program instructions does not make a model.

Acknowledgements

Bruce Edmonds is supported as part of the ESRC-funded, UK part of the “ToRealSim” project, grant number ES/S015159/1.

References

Achter, S., Borit, M., Chattoe-Brown, E., Palaretti, C. & Siebers, P.-O. (2019) Cherchez Le RAT: A Proposed Plan for Augmenting Rigour and Transparency of Data Use in ABM. Review of Artificial Societies and Social Simulation, 4th June 2019. https://rofasss.org/2019/06/04/rat/

Arnold, E. (2014). What’s wrong with social simulations?. The Monist, 97(3), 359-377. DOI:10.5840/monist201497323

Baily, M. N., Litan, R. E., & Johnson, M. S. (2008). The origins of the financial crisis. Fixing Finance Series – Paper 3, The Brookings Institution. https://www.brookings.edu/wp-content/uploads/2016/06/11_origins_crisis_baily_litan.pdf

Chattoe-Brown, E. (2018) What is the earliest example of a social science simulation (that is nonetheless arguably an ABM) and shows real and simulated data in the same figure or table? Review of Artificial Societies and Social Simulation, 11th June 2018. https://rofasss.org/2018/06/11/ecb/

Edmonds, B. (2001) The Use of Models – making MABS actually work. In. Moss, S. and Davidsson, P. (eds.), Multi Agent Based Simulation, Lecture Notes in Artificial Intelligence, 1979:15-32. http://cfpm.org/cpmrep74.html

Edmonds, B. (2020) Basic Modelling Hygiene – keep descriptions about models and what they model clearly distinct. Review of Artificial Societies and Social Simulation, 22nd May 2020. https://rofasss.org/2020/05/22/modelling-hygiene/

Edmonds, B., le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root H. & Squazzoni. F. (2019) Different Modelling Purposes. Journal of Artificial Societies and Social Simulation, 22(3):6. http://jasss.soc.surrey.ac.uk/22/3/6.html.

Epstein, J. M. (2008). Why model?. Journal of artificial societies and social simulation, 11(4), 12. https://jasss.soc.surrey.ac.uk/11/4/12.html

Hartmann, S. (1997): Modelling and the Aims of Science. In: Weingartner, P. et al (ed.) : The Role of Pragmatics in Contemporary Philosophy: Contributions of the Austrian Ludwig Wittgenstein Society. Vol. 5. Wien und Kirchberg: Digi-Buch. pp. 380-385. https://epub.ub.uni-muenchen.de/25393/

Krugman, P. (2009) How Did Economists Get It So Wrong? New York Times, Sept. 2nd 2009. https://www.nytimes.com/2009/09/06/magazine/06Economic-t.html

Kuhn, T.S. (1962) The Structure of Scientific Revolutions. Chicago: University of Chicago Press.

Lakoff, G. (1987) Women, fire, and dangerous things. University of Chicago Press, Chicago.

Morgan, M. S., & Morrison, M. (1999). Models as mediators. Cambridge: Cambridge University Press.

Moss, S. (1998) Social Simulation Models and Reality: Three Approaches. Centre for Policy Modelling  Discussion Paper: CPM-98-35, http://cfpm.org/cpmrep35.html

Popper, K. (1957). The poverty of historicism. Routledge.

Vranckx, An. (1999) Science, Fiction & the Appeal of Complexity. In Aerts, Diederik, Serge Gutwirth, Sonja Smets, and Luk Van Langehove, (eds.) Science, Technology, and Social Change: The Orange Book of “Einstein Meets Magritte.” Brussels: Vrije Universiteit Brussel; Dordrecht: Kluwer., pp. 283–301.

Wartofsky, M. W. (1979). The model muddle: Proposals for an immodest realism. In Models (pp. 1-11). Springer, Dordrecht.

Zeigler, B. P. (1976). Theory of Modeling and Simulation. Wiley Interscience, New York.

Edmonds, B. (2022) The Poverty of Suggestivism – the dangers of "suggests that" modelling. Review of Artificial Societies and Social Simulation, 28th Feb 2022. https://rofasss.org/2022/02/28/poverty-suggestivism

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

Some Philosophical Viewpoints on Social Simulation

Leave a comment

By Bruce Edmonds

How one thinks about knowledge can have a significant impact on how one develops models as well as how one might judge a good model.

  • Pragmatism. Under this view a simulation is a tool for a particular purpose. Different purposes will imply different tests for a good model. What is useful for one purpose might well not be good for another – different kinds of models and modelling processes might be good for each purpose. A simulation whose purpose is to explore the theoretical implications of some assumptions might well be very different from one aiming to explain some observed data. An example of this approach is (Edmonds & al. 2019).
  • Social Constructivism. Here knowledge about social phenomena (including simulation models) are collectively constructed. There is no other kind of knowledge than this. Each simulation is a way of thinking about social reality and plays a part in constructing it so. What is a suitable construction may vary over time and between cultures etc. What a group of people construct is not necessarily limited to simulations that are related to empirical data. (Ahrweiler & Gilbert 2005) seem to take this view but this is more explicit in some of the participatory modelling work, where the aim is to construct a simulation that is acceptable to a group of people, e.g. (Etienne 2014).
  • Relativism. There are no bad models, only different ways of mediating between your thought and reality (Morgan 1999). If you work hard on developing your model, you do not get a better model, only a different one. This might be a consequence of holding to an Epistemological Constructivist position.
  • Descriptive Realism. A simulation is a picture of some aspect of reality (albeit at a much lower ‘resolution’ and imperfectly). If one obtains a faithful representation of some aspect of reality as a model, one can use it for many different purposes. Could imply very complicated models (depending on what one observes and decides is relevant), which might themselves be difficult to understand. I suspect that many people have this in mind as they develop models, but few explicitly take this approach. Maybe an example is (Fieldhouse et al. 2016).
  • Classic Positivism. Here, the empirical fit and the analytic understanding of the simulation is all that matters, nothing else. Models should be tested against data and discarded if inadequate (or they compete and one is currently ahead empirically). Also they should be simple enough that they can be thoroughly understood. There is no obligation to be descriptively realistic. Many physics approaches to social phenomena follow this path (e.g Helbing 2010, Galam 2012).

Of course, few authors make their philosophical position explicit – usually one has to infer it from their text and modelling style.

References

Ahrweiler, P. and Gilbert, N. (2005). Caffè Nero: the Evaluation of Social Simulation. Journal of Artificial Societies and Social Simulation 8(4):14. http://jasss.soc.surrey.ac.uk/8/4/14.html

Edmonds, B., le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root H. and Squazzoni. F. (in press) Different Modelling Purposes. Journal of Artificial Societies and Social Simulation, 22(3):6. http://jasss.soc.surrey.ac.uk/22/3/6.html.

Etienne, M. (ed.) (2014) Companion Modelling: A Participatory Approach to Support Sustainable Development. Springer

Fieldhouse, E., Lessard-Phillips, L. and Edmonds, B. (2016) Cascade or echo chamber? A complex agent-based simulation of voter turnout. Party Politics. 22(2):241-256. DOI:10.1177/1354068815605671

Galam, S. (2012) Sociophysics: A Physicist’s modeling of psycho-political phenomena. Springer.

Helbing, D. (2010). Quantitative sociodynamics: stochastic methods and models of social interaction processes. Springer.

Morgan, M. S., Morrison, M., & Skinner, Q. (Eds.). (1999). Models as mediators: Perspectives on natural and social science (Vol. 52). Cambridge University Press.

Edmonds, B. (2019) Some Philosophical Viewpoints on Social Simulation. Review of Artificial Societies and Social Simulation, 2nd July 2019. https://rofasss.org/2019/07/02/phil-view/

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)

Content

A bad assumption: a simpler model is more general

1 Comment

By Bruce Edmonds

If one adds in some extra detail to a general model it can become more specific — that is it then only applies to those cases where that particular detail held. However the reverse is not true: simplifying a model will not make it more general – it is just you can imagine it would be more general.

To see why this is, consider an accurate linear equation, then eliminate the variable leaving just a constant. The equation is now simpler, but now will only be true at only one point (and only be approximately right in a small region around that point) – it is much less general than the original, because it is true for far fewer cases.

This is not very surprising – a claim that a model has general validity is a very strong claim – it is unlikely to be achieved by arm-chair reflection or by merely leaving out most of the observed processes.

Only under some special conditions does simplification result in greater generality:

  • When what is simplified away is essentially irrelevant to the outcomes of interest (e.g. when there is some averaging process over a lot of random deviations)
  • When what is simplified away happens to be constant for all the situations considered (e.g. gravity is always 9.8m/s^2 downwards)
  • When you loosen your criteria for being approximately right hugely as you simplify (e.g. mover from a requirement that results match some concrete data to using the model as a vague analogy for what is happening)

In other cases, where you compare like with like (i.e. you don’t move the goalposts such as in (3) above) then it only works if you happen to know what can be safely simplified away.

Why people think that simplification might lead to generality is somewhat of a mystery. Maybe they assume that the universe has to obey ultimately laws so that simplification is the right direction (but of course, even if this were true, we would not know which way to safely simplify). Maybe they are really thinking about the other direction, slowly becoming more accurate by making the model mirror the target more. Maybe this is just a justification for laziness, an excuse for avoiding messy complicated models. Maybe they just associate simple models with physics. Maybe they just hope their simple model is more general.

References

Aodha, L. and Edmonds, B. (2017) Some pitfalls to beware when applying models to issues of policy relevance. In Edmonds, B. & Meyer, R. (eds.) Simulating Social Complexity – a handbook, 2nd edition. Springer, 801-822.

Edmonds, B. (2007) Simplicity is Not Truth-Indicative. In Gershenson, C.et al. (2007) Philosophy and Complexity. World Scientific, 65-80.

Edmonds, B. (2017) Different Modelling Purposes. In Edmonds, B. & Meyer, R. (eds.) Simulating Social Complexity – a handbook, 2nd edition. Springer, 39-58.

Edmonds, B. and Moss, S. (2005) From KISS to KIDS – an ‘anti-simplistic’ modelling approach. In P. Davidsson et al. (Eds.): Multi Agent Based Simulation 2004. Springer, Lecture Notes in Artificial Intelligence, 3415:130–144.

Edmonds, B. (2018) A bad assumption: a simpler model is more general. Review of Artificial Societies and Social Simulation, 28th August 2018. https://rofasss.org/2018/08/28/be-2/

© The authors under the Creative Commons’ Attribution-NoDerivs (CC BY-ND) Licence (v4.0)