Is the Hiding Hand Benevolent or Malevolent?

By Bent Flyvbjerg and Cass Sunstein

For references and a longer version of the article, see: https://bit.ly/1hCsaOg 

Albert O. Hirschman loved surprises, ironies, and paradoxes. He was delighted by human foibles and even more, he celebrated human creativity. He was fascinated by what he called "petites idées" and distrusted large claims and law-like generalizations, especially as a basis for policy. He enjoyed serendipity. He insisted that human history provides “stories, intricate and often nonrepeatable,” which “look more like tricks history has up its sleeve than like social-scientific regularities, not to speak of laws.” A refugee from Berlin during the rise of Nazism, he was keenly interested in “the many might-have-beens of history,” including “felicitous and surprising escapes from disaster.”

Despite his distrust of general social-scientific laws, Hirschman came up with quite a few large ideas of his own. One of these is the principle of the Hiding Hand, which is the cornerstone of his classic book, Development Projects Observed, first published in 1967 and recently reissued as a Brookings Classic (Hirschman 1967, 2015).* The Hiding Hand turns out to be a bit of a trick up history’s sleeve. It also provides a felicitous escape from disaster. It’s a story, and an intricate one, but in Hirschman’s view, it is repeatable. Hirschman believes that it tells us a great deal about development, if we are careful to specify the underlying mechanisms.

In Hirschman’s account, social planners tend to be unrealistically optimistic, especially in underdeveloped nations. Ironically, that is fortunate, because if they were more realistic, they would not be bold enough to get started in the first place. Planners begin their projects by greatly overestimating some factor or condition that is indispensable to success, and underestimate difficulties and costs. According to Hirschman, planners thus tend to blunder in a predictable direction, because they neglect “a set of possible and unsuspected threats” to the profitability and even the ultimate existence of their projects. There is an evident connection here with the planning fallacy, much emphasized by behavioral scientists (Buehler 1994, Kahneman 2011), which suggests that people systematically underestimate the time that it will take to complete projects. To this point, Hirschman’s argument can be seen as a version of the planning fallacy writ large.

In Hirschman’s account, social planners tend to be unrealistically optimistic. Ironically, that is fortunate, because if they were more realistic, they would not be bold enough to get started in the first place. 

Fortunately, the planners’ neglect of bad surprises is countered by a much happier surprise, which involves the sheer power of human creativity. Planners do not merely overestimate the likelihood of success and underestimate costs; they also underestimate potential responses to failure. Once things begin to go wrong, people discover unexpected ways to set them right, according to Hirschman. Hence the idea of a Hiding Hand, which “beneficially hides difficulties from us” and thus renders them invisible. The oddity is that while planners might never have authorized certain projects if they had had an accurate sense of the obstacles and costs that those projects would encounter, the result of the Hiding Hand is to produce an outcome that is as good as what the planner originally thought—or perhaps even better. This benevolent outcome is secured by what Hirschman called "providential ignorance" (Alacevich 2014: 157).

Hirschman is far too careful to insist that whenever planners overestimate the likelihood of success, the Hiding Hand will come to the rescue. He was well aware that projects exist, "from bankruptcies and white elephants to lost or ruinously won wars," for which underestimated difficulties and costs were not offset by even larger underestimates of creativity and benefits, but were instead exacerbated by visionlessness and benefit shortfalls (Hirschman 2015: 30). Thus the Benevolent Hiding Hand, which is Hirschman’s topic, has an evil twin, the Malevolent Hiding Hand, which also hides obstacles and difficulties, but in situations in which creativity does not emerge, or emerges too late, or cannot possibly save the day. One of the fiendish acts of the Malevolent Hiding Hand is that it hides not only the initial obstacles and difficulties, but also the barriers to creativity itself. Indeed, Hirschman’s own emphasis on the benevolence of the Hiding Hand might well serve (and perhaps has served) to empower the Malevolent Hiding Hand. For such situations, Streeten (1984: 116) talks about the "Principle of the Hiding Fist." Picciotto (1994b: 302) similarly observes that there might be "two hiding hands." One is applied by decisionmakers in the right situation, the other in the wrong one. In the latter case, "they [decisionmakers and their projects] would ... sink," concludes Picciotto.

The basic mechanisms driving the malevolent Hiding Hand are ignorance, psychology, and power (Flyvbjerg 2009). Ignorance points to the knowledge problem faced by planners of all kinds, even the most well-motivated (Hayek, 1945) and in particular to the difficulty of anticipated unintended consequences and systemic effects (Dorner, 1999). By itself, psychology might provide a sufficient explanation of the Malevolent Hiding Hand (as behavioral scientists have suggested, see Kahneman 2011). On that account, initial optimism is again an issue, but under the Malevolent Hiding Hand, such optimism applies to both the estimation of difficulties/costs and of creativity/benefits, whereas for the Benevolent Hiding Hand optimism applies to difficulties/costs but pessimism to creativity/benefits. For the Malevolent Hiding Hand, difficulties and costs therefore get optimistically underestimated, whereas creativity and benefits get just as optimistically overestimated. This double optimism at the outset comes back to haunt the project during delivery as a double whammy of cost overruns, delays, and other unanticipated hardships compounded by benefit shortfalls. Here is the planning fallacy writ very large, i.e., applying to not only schedule, and not only to initial obstacles (as Hirschman saw) but to costs and benefits in the widest sense. In the regulatory context, it is sometimes suggested, with confidence, that initial estimates of costs will turn out to be greatly overestimated, because regulated entities innovate and hence drive costs down. But sometimes such innovation does not occur, and initial estimates were actually far too optimistic (Sunstein, 2013).

Hirschman glossed over the issue and basically seduced his readers into believing him, through storytelling and what Paul Krugman disapprovingly calls the "richness of plain English."

Where optimism is innocent and unintentional, power play is deliberate and calculated. Planners who use power to activate the malevolent Hiding Hand deliberately underestimate difficulties/cost and overestimate creativity/benefits. They do this to make their projects look good on paper, which they see as increasing their chances of getting their projects approved and funded. Funding typically happens in tough competition with other projects in a tight budgetary process, leading to agency behavior and moral hazard. Advocates of regulation sometimes act similarly, underestimating costs and overestimating benefits in order to promote their goals. There might well be a thin line between deliberate underestimates/overestimates and motivated reasoning (Redlawsk, 2002), which can lead planners, and those who support their efforts, sincerely to believe in assessments that fit with their own hopes and commitments. We suspect that motivated reasoning often plays a large role and that there is a grey area between optimism and power where the two may blend and it is not always clear which is which, even to the actors involved.

It would be extravagant to insist that it is generally or universally good for planners to underestimate difficulties on the ground that people will discover inventive and unanticipated ways to solve those difficulties. Officials who are considering new regulations, knowing that compliance would be extremely expensive or impossible, ought not to proceed on the ground that technological innovation will inevitably make compliance inexpensive or feasible – even though impressive environmental innovation has sometimes occurred in the past. An uncharitable reading of Hirschman's work on the Hiding Hand would suggest that he has committed an identifiable error, which social scientists call “sampling on the dependent variable.” Suppose, for example, that we wanted to understand what makes for a successful entrepreneur, and that we decided to find out by studying a set of successful entrepreneurs. Suppose we learned that the vast majority of them are exceedingly optimistic. From that finding, it would be a mistake to conclude that optimism is a necessary or sufficient condition for entrepreneurial success. There are a lot of failed entrepreneurs out there, and maybe most of them were exceedingly optimistic too. Maybe that trait, even if shared by the successful entrepreneurs, has no causal relationship to their success.

Hirschman identifies some striking instances of a Benevolent Hiding Hand, but his sample size is very small—only 11 projects—and his results are therefore open to random factors. There is little doubt that for countless unsuccessful development projects, the Hiding Hand did not work so well, or turned out to be malevolent, because the blindness at the initial stage is not countered by unanticipated creativity later on. True, and importantly, optimistic planners are sometimes rescued by such creativity, but much of the time, creativity is not triggered, and even if it is, it is not nearly enough to rescue their projects. In such instances, ignorance turned out not to be providential but inopportune instead. We could easily imagine an impressive if somewhat downbeat book, perhaps with the same title as Hirschman's classic, that catalogs a set of failures, bred by a failure to foresee obstacles and challenges that confound planners of many sorts. Indeed, it is not necessary to exercise our imaginations. James Scott’s (1999) wild and brilliant book, Seeing Like a State, is merely the best example.

Hirschman did not, of course, produce such a book, and we do not believe that writing it would have much interested him. While he liked human foibles, he was delighted not by blunders and failures, but by history’s generous tricks, by serendipity and silver linings, and perhaps above all by “felicitous and surprising escapes from disaster.” He was no romantic, but he preferred happy endings, and he firmly believed that the Benevolent Hiding Hand "typically" applies, that is, in more cases than not, and that it was therefore the more overarching and more interesting principle for a general understanding of economic development and project behavior (Hirschman 2015: 1, 13). But with his small sample of 11 projects, Hirschman was in no position to establish whether this belief could be empirically substantiated, or which of the two Hiding Hands was the more prevalent. Hirschman glossed over the issue and basically seduced his readers into believing him, through storytelling and what Krugman (1994: 287) disapprovingly calls the "richness of plain English." (We suspect that like any good storyteller, Hirschman seduced not only his readers but also himself.)

The question therefore remains whether the Benevolent Hiding Hand or the Malevolent Hiding Hand is the more prevalent in policy and practice. Hirschman never answered this question. We do, here: https://bit.ly/1hCsaOg.