The Benefits of Being Unrealistic

By Adam Elkus

As a PhD student in computational social science at George Mason University, I have biases that are understandable. I find that traditional social science methodologies downplay complexity, interaction, path dependence, randomness, network effects, emergence, nested social relations, multiple equilibria, cycles, and heterogeneity. They ignore bounded rationality and cognitive realism. The list goes on. However, I’m also going to speak up here about the benefits of being “unrealistic” — and why social science modelers ought to maintain a diverse toolkit.

The phrase “all models are wrong” and “models are maps” (not the territory) come to mind, but we can do better than that. Ultimately, what using a model as a map implies is that we are using the model as a stylized representation of reality. I’m reminded here about the controversy over Michael Chwe’s recent book on Jane Austen and game theory — Jane Austen didn’t intend to have her characters behave like John Nash or R.A. Fisher. It’s convenient for us to represent Jane Austen books as games, much in the same way its convenient to represent Viking societies with graph theory. It’s an imposition of our own, and we can at least hope that we don’t do too much violence to what we’re trying to represent. Often times, however, we do exactly that:

In settings like biology, medicine — or even more ambitious: social sciences — there is no underlying analytic theory. Although we might call some parameters of the model by the same name as some things we measure experimentally, the theory we use to inform our measurement is not related to the theory used to generate our model. These sort of models are heuristics. This means that when a heuristic model is wrong, we don’t really know why it is wrong, how to quantify how wrong it will be (apart from trial and error), or how to fix it. Further, even when heuristic models do predict accurate outcomes, we have no reason to believe that the hidden mechanism of the model is reflective of reality. Usually the model has so many free-parameters, often in the form of researcher degrees of freedom, that it could have been an accidental fit. This is often a concern in ecological modeling where we have to worry about overdetermination.

So we can dispense with the idea that a model that makes more realistic assumptions in some areas we know about (cognition, rationality, social interaction, heterogeneity, feedback) is necessarily going to be more useful to us than an older model that makes less realistic assumptions in those same areas (such as representing model agents as aggregates in system dynamics). We are usually making other unrealistic assumptions in areas of equal importance merely by creating a model in the first place. There is nothing inherent in iterating the Prisoner’s Dilemma and adding genetic algorithms with tournament-style selection, for example, that necessarily makes it more realistic than a traditional one-shot PD.

It’s obvious that the question you want to answer and the kind of answer you want to get should dictate choice of methods. But it’s also less obvious how this should impact choice of how fine-grained a model you need. Such a distinction often leads to the charge of “___ is unrealistic!” (usually referring to some kind of classical game theory or choice-theoretic assumptions) followed by trotting out of the “models are maps” and “all models are wrong” cliches. While true in a banal sense, defenders of more “unrealistic” assumptions about older modeling techniques surely can make better arguments.

Consider artificial intelligence and formal computer science as disciplines. AI and algorithms have been designed to perform tasks traditionally characteristic of humans — but in ways humans obviously do not solve problems. As Scott Page points out, humans use heuristics to deal with computational problems that scale poorly when analyzed by computers or find “good enough” solutions. But understanding that a problem’s worst-case computational complexity is so daunting in the first place is useful — without it we might not understand why heuristics are being employed or why they work.

This holds true beyond the worlds of AI and analysis of algorithms. We might conclude that a true Weberian monopoly of force is logically impossible — when we say “the state” has a monopoly on force, we ignore the fact that this “monopoly” could be eroded easily if enough elites with control over the means of violence decide it is no longer in their interest to contribute to it. The “state” is, after all, an aggregation of those elites. And in many parts of the world, achieving a preponderance of force (as opposed to a monopoly) is more realistic and historically accurate. But the “ideal type” is useful for modeling (verbally) what a state with true license over lawful organized violence resembles — and measuring how far the empirical world falls from ideal case still is theoretically meaningful.

Second, as a behavioral economics textbook pointed out, merely including more “behavioral” assumptions does not necessarily ensure that an behavioral economics model is going to outperform a standard economic model. The same holds for many other related disciplines. Realistic insights about individual cognition and decision may not scale up well to social aggregates — norms, for example, are far more situational in character for the average individual than they are in society as a whole.

Finally, any subject of interest is bound to be multi-layered. We are not likely to find a way to effectively link all of those layers without either making our model either too complex to be used and properly communicated or too simplistic to be truly comprehensive.

In my own field of strategic theory, military theorists are beginning to understand that there are multiple models of strategic reasoning — because the strategist’s task itself is heterogeneous. Figuring out how to get a large mass of men, machines, and supplies across the Atlantic in time to resist a Soviet invasion is strategy. Using mathematical programming to optimize military production is strategy. Game-theoretic models of coercion and compellence are also strategy. But so are the difficult tasks of managing coalitions, sense making and problem framing in ambiguous situations, and intersubjective learning about the enemy and civilian population.

Historians that can examine a given campaign or operation in granular detail get to weave all of these together into a coherent narrative. But when we begin to talk more generally, we start to face the problem of representation. In my time as an MA student at the Georgetown Security Studies program, I took a class from Kenneth Pollack on Military Analysis. We studied World War II’s European Theater of Operation (ETO), a broad area that concerns campaigns and operations in Europe, Northern Africa, the British Islands, Russia, the Atlantic, and the Strategic Bomber Offensive.

Our task in making an counterfactual analysis of a campaign or operation was to weave together a number of levels and contexts. One had to take into account the role of coalition pressures, industrial and economic supply and production, military training and organization, terrain, feedback effects from other campaigns and operations, and leadership styles to do justice to the topic. But feeding all of that into a model is a recipe for overfitting and poor out of sample predictive value.

Parsimony may be a virtue, but it is also a cruel mistress. All generalization — including qualitative case work — involves parsimony. What “models as maps” rhetoric misses is that the social scientist is actually carrying a suitcase of maps. Instead of one big and coarse map he or she is using to find their way around a metro, they have a dozen maps of varying size, detail, and specification they have to pull out each time they change subway lines. Sometimes one map being more “wrong” will be the key to another being more “right.”