The Batting Average Metaphor in Intelligence Analysis

Source available at the following link: Evaluating the Quality of Intelligence Analysis: By What (Mis) Measure?: Intelligence and National Security: Vol 27 , No 6 - Get Access (tandfonline.com)

"Metaphors from baseball are frequently employed by scholars to frame the evaluation of intelligence performance precisely because many useful inferences can be derived from them.18 For example, the difference between the fielding percentage, where most anything less than perfection is an error, and a batting average, which provides more room for error without condemnation,19 highlights the importance of the standard used to evaluate relative performance. In addition, the use of the batting average metric also makes it clear that it is relative success versus an opposing force in the context of a competition where the fates of the batters will, as Betts says, ‘depend heavily on the quality of the pitching they face’.20 The fact that relative success or failure is contingent on the skill of the opposition has clear parallels in the world of intelligence

Most importantly, however, the batting average metaphor provides us with a notional sliding scale of proficiency. Glenn Hastedt suggests that while ‘accuracy is an intuitively appealing standard, lending itself to an easy to-convey scoreboard counting method to assess the performance’ of intelligence analysts, ‘the use of accuracy as a standard also raises the questions: How accurate must one be? What is an acceptable ‘‘batting average’’?’21 Betts says that while ‘there is a limit to how high the intelligence batting average will get’ because perfection is unattainable, that ‘is not to say that it cannot get significantly better’.22 In 1964 Klaus Knorr suggested that rather than eliminate inaccuracy or surprise a more realistic goal would be ‘to improve the ‘‘batting average’’– say, from 275 to .301’.23 Betts further develops this idea when he suggests that ‘a batter may improve his average by changing his stance or swing, and such changes are worth making even if the improvement is small. Raising a few players’ averages from .275 to .290 is an incremental improvement, but it could turn out to make the difference in whether the team finishes the season in second place or in first’.24 But even if success rates do increase, Betts goes on to say that ‘even a .900 average will eventually yield another big failure’.

The inevitability of big failures can best be illustrated by extending the metaphor to discuss the relative importance or significance of each at bat. In baseball, each at bat in the spring and early summer is less important than it is in the fall. But nothing comes close to the significance of the World Series, particularly the last inning of the seventh game. Yet even then, while not desired failure at those highly significant moments is still normal. It is possible to strike out in the last at bat of the World Series, and great batters have at times had great failures. Or, in the intelligence context, ‘even the best intelligence systems will have big failures’, according to Betts.25

That is not to say that striking out in the last at bat of the World Series is acceptable, though. The batting average metaphor also provides us with implicit normative standards that we can use to evaluate each individual success or failure. As the late CIA officer Stanley Moskowitz pointed out, ‘if the bases are loaded in the bottom of the ninth of the seventh game of the World Series, and our .300 hitter strikes out, you bet fans will say he failed ... or worse’.26 As Betts suggests: ‘a batter who strikes out is certainly at fault for failing to be smarter or quicker than the pitcher. Whether the batter should be judged incompetent depends on how often he strikes out against what quality of pitching. A batter with a .300 average should easily be forgiven for striking out occasionally, while one who hits .150 should be sent back to the minors’.27

One major flaw in the use of the batting average metaphor, though, is that the pre-set normative expectations for performance drawn from baseball may not correspond well to the evaluation of intelligence analysis. In baseball, below .200 is bad and above .350 is very good. So what is the appropriate batting average for intelligence analysis? We do not know. According to Richard Posner, ‘no one has any idea’ of what it actually is or should be.28

Perhaps another metaphor might be useful to introduce here. Another way to illustrate the ratio of success to failure without the pre-set expectation of the batting average analogy is through the use of the glass half-full/glass half empty metaphor. There is a saying that an optimist sees a glass as half-full whereas a pessimist might see the glass as half-empty. If the full part of the glass represents intelligence accuracy (or success), Betts– on seeing that glass– would probably say that it can never be 100% full. Others who highlight analytic accuracy and its role in preventing surprise would suggest that even though the glass might not be full, it is far from empty. For illustrative purposes, this analogy is both more general and more universal than the batting average one, and perhaps easier to use when explaining the issues and problems in evaluating intelligence analysis over time.

But how full is the glass, or what is the ratio of success to failure? And how full should the glass be– or what should the ratio between success and failure be– given the difficulties of intelligence analysis and tradeoffs involved? A judicious exploration of the history of intelligence analysis over time should be able to provide us with answers to these questions

XX

Citations

18For example, see: John Hedley, ‘Learning from Intelligence Failures’, International Journal of Intelligence and Counterintelligence 18/3 (2005) pp.435–50; Paul R. Pillar, ‘Intelligent Design? The Unending Saga of Intelligence Reform’, Foreign Affairs, March/April 2008 (citing Betts); Robert Mandel, ‘Distortions in the Intelligence Decision-Making Process’ in Stephen J. Cimbala (ed.) Intelligence and Intelligence Policy in a Democratic Society (Dobbs Ferry, NY: Transnational Publishers, Inc. 1987) pp.69–83; Mark M. Lowenthal, ‘The Burdensome Concept of Failure’ in Alfred C. Maurer, Marion D. Turnstall, and James M. Keagle (eds.) Intelligence: Policy and Process (Boulder, CO: Westview 1985) pp.43–56; Mark M. Lowenthal, ‘Towards a Reasonable Standard for Analysis: How Right, How Often on Which Issues?’ Intelligence and National Security 23/3 (2008) pp.303–15.

19As Robert Jervis points out, ‘if we were right something like one time out of three we would be doing quite well’. Robert Jervis, ‘Improving the Intelligence Process: Informal Norms and Incentives’ in Alfred C. Maurer, Marion D. Tunstall and James M. Keagle (eds.) Intelligence: Policy and Process (Boulder, CO: Westview Press 1985) p.113.

20Betts, Enemies of Intelligence, p.65

21Glenn Hastedt, ‘Intelligence Estimates: NIEs vs. the Open Press in the 1958 China Straits Crisis’, International Journal of Intelligence and CounterIntelligence 23/1 (2009) pp.104–32.

22Richard K. Betts, ‘Fixing Intelligence’, Foreign Affairs 81/1 (2002) p.59.

23Klaus E. Knorr, ‘Failures in National Intelligence Estimates: The Case of the Cuban Missiles’, World Politics 16/3 (1964) p.460.

24Betts, Enemies of Intelligence, p.186

25Betts, ‘Fixing Intelligence’, p.59.

26Stanley Moskowitz, ‘Intelligence in Recent Public Literature: Uncertain Shield: The U.S. Intelligence System in the Throes of Reform by Richard A. Posner’, Studies in Intelligence 50/ 3 (2006), 5https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/ csi-studies/studies/vol50no3/Uncertain_Shield_7.htm4 (accessed 21 June 2012)

27Betts, Enemies of Intelligence, pp.185–6.

28Richard A. Posner, Preventing Surprise Attacks: Intelligence Reform in the Wake of 9/11 (Lanham, MD: Rowman and Littlefield Publishers, Inc. 2005) p.106. Kuhns made a similar point when he answered the question ‘just how bad is it out there?’ by saying ‘the short answer is, no one knows.’ Woodrow J. Kuhns, ‘Intelligence Failures: Forecasting and the Lessons of Epistemology’ in Richard K. Betts and Thomas G. Mahnken (eds.) Paradoxes of Strategic Intelligence: Essays in Honor of Michael I. Handel (London: Frank Cass Publishers 2003), p.82