What The Fox Doesn’t Know?

By Adam Elkus

There’s an emerging dustup between FiveThirtyEight‘s Nate Silver and The New Republic’s Leon Wieseltier. Since much of this already boiling down to a re-hash of CP Snow’s “Two Cultures” debate, I’ll try to look at each’s argument and then observe some strengths and flaws. TL: DR — both are talking past each other, one has some big flaws, and the other is missing the point.

First, Silver. Reading through Silver’s blog, I see two sorts of arguments being made from the philosophy of science that Silver (perhaps in the interest in readability, doesn’t fully explain) — prediction and falsification. Silver sees the primary problem with journalism as being one in which a gap exists between collection and organization of information and explanation and generalization. Silver’s idea of how to fix this gap is strongly bound up in the idea of producing knowledge that is both falsifiable and have good out of sample predictive qualities.

For example, they cite three factors they say were responsible for Mitt Romney’s decline in the polls in early mid-September: the comparatively inferior Republican convention, Romney’s response to the attacks in Benghazi, Libya, and Romney’s gaffe-filled trip to London. In fact, only one of these events had any real effect on the polls: the conventions, which often swing polls in one direction or another. (This does not require any advanced analysis — it’s obvious by looking at the polls immediately before and after each event.) Explanation is more difficult than description, especially if one demands some understanding of causality. …. …But while individual facts are rigorously scrutinized and checked for accuracy in traditional newsrooms, attempts to infer causality sometimes are not, even when they are eminently falsifiable.

Explanation is about why particular things occur, and these explanations should ideally be falsifiable. Notice that Silver does not necessarily say that all explanations are falsifiable. If he did, this would rule out large swaths of the hard sciences that rely on notions that are not directly falsifiable.  He would also rule out the utility of heuristic understandings of phenomena where good data does not exist, or where the results of statistical meta-analysis are inconclusive and contradictory. Still, Silver seems to privilege explanations that are falsifiable — and as I will later detail — gloss over some of the enormous problems with the conception of science that he mentions as a model for his site.

He later goes on to make a covering-law esque argument that particular explanations should be evaluated for how well they scale with the aim of finding useful general truths. He equates explanation and causality with the classical model of an explanandum to be explained and a set of premises that explain it. Silver says that a generalization must be tested by how well it predicts out of sample, and equates this to falsification in the absence of laboratory experiments. However, while Silver may have a point about prediction, there are some distinct nuances to how falsification has been considered in the philosophy of science.

The problem with Silver’s argument is that he glosses over just how hard it is to actually get rid of a theory. If you believe Imre Lakatos, than the hard core of a research program itself is unfalsifiable. If you subscribe to a coherentist view in the philosophy of science, you may believe (like Duhem-Quine) that a theory is not one thing but a web and one has to defeat the core of theory and its outlying components. You may not, as per Feyeraband, believe that we can rise to a general model of science and that domain-specific principles rule. And this is to say nothing of the vast array of historical and sociological work on the ways in which science is actually practiced, which to some extent have some uncomfortable aspects in common with Silver’s critique of punditry as being driven by strong ideological priors.

Now, if we focus solely on the aspect of predictive accuracy Silver seems to be on stronger grounds. Given that it is so hard to really falsify a theory, and that it is also easy to rescue a theory by saving it from failures to predict, Milton Friedman made a much-maligned argument that theory itself is inherently tautological and what matters is whether or not the theory accounts for things that haven’t been observed yet:

The ultimate goal of a positive science is the development of a “theory” or, “hypothesis” that yields valid and meaningful (i.e., not truistic) predictions about phenomena not yet observed. Such a theory is, in general, a complex intermixture of two elements. In part, it is a “language” designed to promote “systematic and organized methods of reasoning.” In part, it is a body of substantive hypotheses designed to abstract essential features of complex reality. Viewed as a language, theory has no substantive content; it is a set of tautologies. Its function is to serve as a filing system for organizing empirical material and facilitating our understanding of it; and the criteria by which it is to be judged are those appropriate to a filing system. Are, the categories clearly and precisely defined? Are they exhaustive? Do we know where to file each individual, item, or is there considerable ambiguity? Is the system of headings and subheadings so designed that we can quickly find an item we want, or must we hunt from place to place? Are the items we shall want to consider jointly filed together? Does the filing system avoid elaborate cross-references?

Friedman in many ways bypasses the problem of falsification by noting that a theory’s internal consistency is not necessarily important because consistency can easily lapse into tautology:

A hypothesis is important if it “explains” much by little, that is, if it abstracts the common and crucial elements from the mass of complex and detailed circumstances surrounding the phenomena to be explained and permits valid predictions on the basis of them alone. To be important, therefore, a hypothesis must be descriptively false in its assumptions; it takes account of, and accounts for, none of the many other attendant circumstances, since its very success shows them to be irrelevant for the phenomena to be explained. To put this point less paradoxically, the relevant question to ask about the “assumptions” of a theory is not whether they are descriptively “realistic,” for they never are, but whether they are sufficiently good approximations for the purpose in hand. And this question can be answered only by seeing whether the theory works, which means whether it yields sufficiently accurate predictions. The two supposedly independent tests thus reduce to one test.

For Friedman the issue is that whether or not valid predictions follow from the minimal components of a theory that can approximate something of interest. This actually contradicts the Tetlock-like argument that Silver makes about ideologically strong priors held by pundits. A pundit could believe any number of things that might seem patently ridiculous — but what matters is that they permit valid predictions. Silver might agree that this is true, and make an argument (as he has) that pundits should be open to revising their beliefs in light of failed predictions, updating their priors in a Bayesian fashion. While I would agree that this would be a Good Thing, it also shows Silver’s lack of understanding about the nature of punditry.

When Silver talks about strong priors and ideological beliefs, he’s in some ways paraphrasing Noah Smith’s now-infamous explanation of “derp” as unusually strong Bayesian belief states that resist posterior estimation. Silver and Smith are arguing that even math-averse pundits have implicit models of how the world works, and those models ought to be evaluated for predictive accuracy. It is true that all pundits that make normative arguments about complicated social things have implicit models of the world, and also make implicit predictions about the future. But this is secondary really to the purpose of punditry to begin with.  Pundits do not see things in terms of probability — Bayesian or Frequentist. The basic column has the following format: “X is the present state of the world, Y is wrong/right in it, Z should be done/not done.” X is the area most amenable to Silver-like data analysis, but as we move from X down to Z the idea of using scientific arguments to address it becomes more and more problematic.  The relationship between science and religion, for example, is still not something that we have gotten a good handle on despite centuries of debate.  Moreover, in most public policy issues data will bound the range of acceptable policy options but not necessarily do much more than that.

Wieseltier’s argument, on the other hand, is a farrago of nonsense. Whereas Silver’s argument simply is problematic because it fails to grapple with some complexities of science and opinion, Wieseltier seems more interested in rhetoric than anything else:

He dignifies only facts. He honors only investigative journalism, explanatory journalism, and data journalism. He does not take a side, except the side of no side. He does not recognize the calling of, or grasp the need for, public reason; or rather, he cannot conceive of public reason except as an exercise in statistical analysis and data visualization. He is the hedgehog who knows only one big thing. And his thing may not be as big as he thinks it is. Since an open society stands or falls on the quality of its citizens’ opinions, the refinement of their opinions, and more generally of the process of opinion-formation, is a primary activity of its intellectuals and its journalists. In such an enterprise, the insistence upon a solid evidentiary foundation for judgments—the combating of ignorance, which is another spectacular influence of the new technology—is obviously important. Just as obviously, this evidentiary foundation may include quantitative measurements; but only if such measurements are appropriate to the particular subject about which a particular judgment is being made. The assumption that it is appropriate to all subjects and all judgments—this auctoritas ex numero—is not at all obvious. Many of the issues that we debate are not issues of fact but issues of value. There is no numerical answer to the question of whether men should be allowed to marry men, and the question of whether the government should help the weak, and the question of whether we should intervene against genocide. And so the intimidation by quantification practiced by Silver and the other data mullahs must be resisted. Up with the facts! Down with the cult of facts!

First, the question is posed wrongly as a matter of measurement and fact. The specific criticism of punditry that Silver makes is one that pundits do not revise their beliefs after events cast doubt on the accuracy of a belief to predict future events. Say that John Mearsheimer, in making an normative policy argument for realist policies, argues that the international system has certain rules and thus himself argues that those rules will lead to certain outcomes. It is fair for Phillip Schrodt to highlight the failure of the system to behave in the way he says, and argue that this should have implications for whether we rely on his theory. Silver’s error is in the assumption that beliefs are predictions, as opposed to the sensible observation that strong beliefs will usually have predictive implications.  Certainly numbers cannot decide the issue of whether men should marry men, but if arguments against same-sex marriage warn that more liberal attitudes towards homosexuality will lead to the decline of marriage it is fair for Silver to try to see if this belief accounts for the variation in marriage and divorce.  It is precisely the fact that internally consistent beliefs can be tautological, as Friedman observes, that makes prediction useful.

Second, nowhere does Silver say that data ought to decide normative issues. The strongest statement that Silver makes about this in his manifesto is ironically counter to the image the TNR casts of him as a quant expressing a view from nowhere: Silver argues that scientific objectivity is distinct from journalistic objectivity in that it should make statements about whether certain arguments can be factually sustained. This is not necessarily an argument that empiricism should be the final arbiter, but that it ought to make a statement about what truths can be discerned from investigation about the rightness and wrongness of argument. And it is also not too much different from the notion of journalistic objectivity, as Silver argues. A good journalist doesn’t represent all of the sides of an issue, they give the reader information as to which ones are problematic. I am not sure, again, how he can square the circle between two notions — it is one thing to scientifically evaluate competing hypotheses, another to scientifically evaluate competing normative beliefs that do not really take the form of hypothesis or theory (even if they may have implicit hypotheses and theories embedded).

Wieseltier gives away his real problem with Silver when he notes this:

The intellectual predispositions that Silver ridicules as “priors” are nothing more than beliefs. What is so sinister about beliefs? He should be a little more wary of scorning them, even in degraded form: without beliefs we are nothing but data, himself included, and we deserve to be considered not only from the standpoint of our manipulability. I am sorry that he finds George Will and Paul Krugman repetitious, but should they revise their beliefs so as not to bore him? Repetition is one of the essential instruments of persuasion, and persuasion is one of the essential activities of a democracy. I do not expect Silver to relinquish his positivism—a prior if ever there was one—because I find it tedious.

It were one thing if punditry consisted of abstract deduction. But it does not. Punditry is about persuasion. Pundits do not make logical arguments from first principles or write mathematical proofs. Nor do pundits utilize any of the techniques of logic found in mathematics and philosophy, write sound mathematical definitions, or build their arguments off of logical deductions in the way that all mathematicians must work off previously proved things. Instead, Wieseltier is making a strong argument that “persuasion is one of the essential activities of a democracy.” Hence Will and Krugman should be free to repeat their beliefs for dramatic effect, in the hope that it would persuade others that they are right. This contradicts Wieseltier’s earlier arguments about reason, logic, and deduction. If Wieseltier wants to argue a reason-based defense of the humanities, which I do find persuasive, he cannot have it both ways. Public reason and persuasion are not the same thing — taken to one extreme persuasion becomes sophistry. 

Sophistry, however, is what Wieseltier has been selling for a very long time. In arguing for his policy positions — particularly on the Iraq War. Wieseltier’s columns at TNR present no deductively rigorous argument on the question of intervention and America’s place in the world. Instead they are extended fits of moral posturing, in which he constantly exhorts the reader to a titanic struggle against evil. Instead of logical and rigorous arguments about whether or not a particular stance on Ukraine follows from a particular train of logic, Wieseltier’s world instead is a emotionally charged trip into glory, courage, and justice — where every struggle is always Munich, and every politician an inferior shadow of a Churchillian figure exhorting the populace into total mobilization. Wieseltier, in other words, is engaging in a particularly sophistic form of persuasion that aims to convince us that we ought to embrace a position of total mobilization by utilizing rhetoric and repetition. Indeed, Matt Yglesias (who has an undergraduate degree in philosophy) got it right when he flagged a somewhat muddled take on Kant by Wieseltier — Wieseltier is TNR’s book reviews editor. He is a literary scholar, not a philosopher.  I certainly know I have not lived up to the standards that I am holding Wieseltier to in my own writings, but I at least have become acutely aware that there is something wrong with the kind of argumentative style that I sometimes fall into. Wieseltier, however, conflates public reason with emotive rhetoric.

I must admit that I have my own doubts about Silver’s new enterprise. And like a Bayesian, I have a prior belief that I will adjust when the “data” comes in to evaluate it. I do not feel entirely comfortable with the arguments he makes and also am skeptical that data without mechanisms or heuristic understanding will really deliver the insights that the site promises. That being said Silver strikes me as a very smart person who has thought very deeply about the problems with modern journalism. I at least feel somewhat confident that he will be an evolutionary improvement over the existing model. Wieseltier, however, is the very symbol of the kind of pundit that makes even the most hyperbolic Silver critiques seem understandable. I will take data enthusiasm over Wieseltier’s “persuasion” any day of the week.  I do not also think that Silver will crowd out “public reason.” Indeed, the popularity of Nassim Nicholas Taleb — a quant turned philosopher — seems to indicate otherwise. Someone like Taleb, who grounds arguments in the style of a mathematician or philosopher rather than a statistician (and unlike Wieseltier has a body of technical work that can be philosophically evaluated) will be first to check a Silver-like data journalist if they overreach. We need both empiricists and rigorous deductive analysts, and ideally combinations of both.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s