Using Brown M&Ms to better inform decisions on the battlefield: Some thoughts on Survey Vetting

As we remove “boots on the ground” from places like Afghanistan, the Philippines, and the Horn of Africa, commanders must increasingly rely on alternate data points to inform assessments. Public Perception data is one of the data sources we typically lean on. I think most ORSAs, particularly those who have deployed in support of OIF/OEF, have experience using public perception survey data in some form or another. I’ve spent the last three years, both at the Center for Army Analysis as well as at the Gallup Organization, working with international public perception data. As many of us likely know, contracting survey administration out to third party foreign vendors is not without challenges. There are inevitably questions about the validity and quality of survey data. How can we know the surveys commanders rely on to inform decisions were not filled out in mass in some basement in Kabul? There are cases, particularly in Afghanistan, where an analyst can look at different surveys asking similar questions, during the same time period and location, and come to drastically different conclusions. Obviously, this may cause one to question the usefulness of survey data entirely. In fact, many decision makers and analysts may have already discounted the value of battlefield surveys. So how can we improve the collection, oversight, and vetting of survey data, particularly as we draw down our manpower and resources on the ground? One solution is to have interviews use smart phones, tablets, or other GPS devices that automatically geocode and timestamp individual surveys. Typically, concerns about cost, interviewer safety, and respondent confidentiality are cited barriers that limit our ability to do this. Admittedly, we may not yet be to the point where we can implement a technology solution in many of the third world counties we operate. As an alternative, I suggest we use something much simpler: brown M&Ms.

I’m a big fan of the Freakonomics series. I’ve been reading, listening to, and watching Steven Levitt and Stephan Dubner for several years now. Their latest book, “Think Like a Freak” has a fascinating chapter which they titled “What Do King Solomon and David Lee Roth Have in Common?” The chapter discusses a bit of game theory and some innovative ways that people have, as Levitt and Dubner phrase it, “taught their garden to weed itself.” The premise is to create or identify some mechanism to trick the guilty into separating themselves from the innocent; therefore revealing their guilt. From the cold women who stood idly while King Solomon drew his sword; threatening to slice a baby in two, to the terrorist who takes out a life insurance policy shortly after the release of the book titled, “SuperFreakonomics: Global Cooling, Patriotic Prostitutes, and Why Suicide Bombers Should Buy Life Insurance,” the book outlines some great examples of successfully employing this strategy. The example that resonated the most to for me was the Van Halen story. Van Halen front man, David Lee Roth, explains in his autobiography, Crazy from the Heat, why the band insisted the promoters remove all brown M&Ms from the backstage area in his autobiography:

Van Halen was the first band to take huge productions into tertiary, third-level markets. We'd pull up with nine eighteen-wheeler trucks, full of gear, where the standard was three trucks, max. And there were many, many technical errors — whether it was the girders couldn't support the weight, or the flooring would sink in, or the doors weren't big enough to move the gear through.

The contract rider read like a version of the Chinese Yellow Pages because there was so much equipment, and so many human beings to make it function. So just as a little test, in the technical aspect of the rider, it would say "Article 148: There will be fifteen amperage voltage sockets at twenty-foot spaces, evenly, providing nineteen amperes . . ." This kind of thing. And article number 126, in the middle of nowhere, was: "There will be no brown M&M's in the backstage area, upon pain of forfeiture of the show, with full compensation."

So, when I would walk backstage, if I saw a brown M&M in that bowl . . . well, line-check the entire production. Guaranteed you're going to arrive at a technical error. They didn't read the contract. Guaranteed you'd run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.

The rock band, Van Halen developed a clever way of separating those promoters who were doing the right thing, from those who were faking the funk. What if we could do something similar with surveys on the battlefield? I would argue that we can, and in most cases, those “brown M&Ms” are already present. There are some obvious things to look for. Almost every survey contract contains the requirement for data entry and the delivery of an SPSS file. One can simply look at the quality of the file to make some preliminary judgments on the vendor. Is the file complete, and correctly coded? Are text fields stored as string values and numbers stored real ones? Are scale, ordinal, and nominal variables properly identified and coded as such? Are missing values and non-responses correctly coded? Are variables consistently named across survey waves? The vendor reveals an awful a lot about themselves in this single file. These initial checks don’t require a statistician, data scientist or any particular expertise. In fact, they can be easily automated so that any individual serving as a contracting officer or contracting officer representative can perform them.

Once this preliminary check of data file quality is complete, one can examine some additional potential “brown M&Ms”. Many of these can be found in the survey metadata. Typically, the survey metadata is stripped out and discarded by the analyst and is rarely analyzed. It does, however, contain several potential indicators of both data and vendor quality. It is standard practice for interviewers doing field work to record the start time and end time of each interview. In many cases the interviewer will also record the interview length. Occasionally, the interview length and the reported start and end times don’t reconcile. As a data analyst, it’s tempting to eliminate this possibility for error by removing or discounting the Interview Length question from the survey questionnaire and simply derive the length from the recorded start and end time. By doing this, however, we lose some valuable information. The inability to perform simple arithmetic may seem trivial, but it sends a signal about interviewer and data entry attention to detail. It also can signal poor vendor quality control mechanisms. Similarly, the distribution of start and end times can be a useful indicator for the analyst. Start and End times that appear to be consistently rounded to the nearest five or ten minute interval should also raise concern. While this could indicate an incredibly consistent and punctual interviewer, it most likely reflects poor attention to detail and quality assurance. These are the low hanging fruit, a few of the easily observed data points within the survey metadata that can help an analyst gage survey data and vendor quality.

What I’ve outlined, much like the “brown M&Ms,” are some of the most basic details that can be easily checked and vetted. If a vendor cannot complete the very simple task of properly coding and vetting an SPSS file, what reason do we have to believe they are completing the monumentally complex task of properly administering a public perception survey in a war zone? Once we’ve done these basic preliminary checks, we can move to more advanced statistical techniques such as cluster analysis and other forms of anomaly detection. This way of thinking, along with a more active statistical approach to survey vetting, will increase the usefulness of public perception survey data and ultimately aid decision makers on the battlefield. Until we do this, and as long as we continue to find “brown M&Ms,” questions about quality will make the value of battlefield public perception data marginal at best.