Gunshot Detection Works, We're Just Not Measuring It Right
As more attention has been paid to the nuances of how policing is done across the country, I’ve seen a significant uptick in discussion about public safety technology. Most of the news articles and opinion pieces speak from a position of ignorance, but that is not necessarily the authors’ or publications’ fault. Law enforcement is hesitant to share details about anything that helps fight crime, which is somewhat reasonable, as there is no clear guidance or understanding as to where the line should be drawn between transparency and operational security. Additionally, the industry partners that support the law enforcement mission are reluctant to release details on their technologies, partially for similar reasons as their customers, but also because of the belief that any perceptions of negative performance will damage sales. In this paper, I will argue that it is in everyone’s best interest to increase the level of transparency on public safety technologies, specifically around efficacy and the recognized return on investment. While the sharing of this information will continue to help build trust with the community, it importantly helps to establish quantifiable and objective reasoning for continuing with any particular public safety investment. To make this point, I will focus on gunfire detection, with an emphasis on ShotSpotter, primarily because the company is mentioned in the media frequently and is often viewed as the market leader.
If you are unaware of what gunshot detection is, basically, it is a microphone that listens for loud noises. When it hears one, some number of algorithms attempt to make a determination that the loud noise is a gunshot, and if so, sends that as an alert to police. I’m being intentionally simplistic here, but in essence, that is the approach that the vast majority of gunshot detection systems operate. Due to this framework, the primary way the various vendors in this industry differentiate themselves is through the quality of the algorithm making the decision. Companies like Safety Dynamics, EAGL Technology, and many others all operate this way. The primary challenge here is that computers just aren’t very good at telling the difference between various types of loud noises. This isn’t an issue in semi-controlled or sparsely populated settings, such as indoors or near the southern border respectively, but in outdoor dense urban environments, there is just too much potential for misclassifications.
Conversely, ShotSpotter has taken a different approach. They use similar algorithms to make an initial recommendation, but they supplement the imperfections in the technology with human analysts1. This human in the loop approach should be considered a model for most AI/ML driven technologies, as it provides a much-needed check against the prevalence of indescribable and unanalyzable computer decision processes. For ShotSpotter, the human adds a bit under 60 seconds of delay2, but the reduction in false positives is massive. Based on my personal experiences, I would estimate that some 90% of all sensor activations are dismissed by the analysts. I’ll note, this isn’t a bad thing. Each of these would have likely been a false alarm if sent directly to police, and competing systems struggle in urban environments for exactly this reason. This efficacy improvement is also what drives costs. ShotSpotter is expensive, much more so than any of their competitors. I personally believe this cost is justified, which is why it is imperative that real metrics are captured, measured, and published, as the current subjective analyses being offered by both law enforcement and industry cannot fully rationalize the significant financial investment.
A frequent refrain from police chiefs is that, if one life is saved, then that should justify the cost3. This isn’t a fair assessment of any technology, as it ignores the opportunity cost. A better question would be to ask, “How many lives could be saved if those monies were spent on something else?” I want to be conscious that I don’t imply a “defund the police” narrative, as this article is arguing for more law enforcement investment, not less, but those funds should be defensible based on real data, and compared to alternate measures. Shockingly, however, most agencies that have implemented ShotSpotter do not record enough detail on alerts to even begin to make rational assessments?. As part of any new public safety technology implementation, not just gunfire detection, it is vital to establish the appropriate policies and procedures to accurately collect data that can be used to make informed judgements on subsequent expenditures. For ShotSpotter in particular, agencies need to collect data on activations, dispatches, evidence recoveries, arrests, associated 911 calls, response times, NIBIN matches, prosecutions, and personnel deployment activities. Without these data, an agency cannot make an objective assessment to the efficacy of the system, so it certainly cannot communicate the same to the public?.
That isn’t to say that collecting all of these data is easy. In fact, agencies should factor in the manpower implications into their procurement decision process. Often, agencies presume there will be a rise in identified shots fired incidents, with a corresponding increase in police dispatches, but the logistical and administrative burdens that the technology will impose are frequently overlooked. This means that the collection of the aforementioned data elements is all too often disregarded, and agency leaders must resort to more subjective and emotional appeals to maintain these systems. But the data are there, if one knows where to look. The NYPD, for example, excels at tracking even the most minute of details, and despite many criticisms, has been fairly forthcoming on the results they have seen?. In particular, the NYPD can provide appropriate attribution to a ShotSpotter alert for each associated arrest, evidence recovery, response time, and subsequent investigation, which is noteworthy when approximately 75% of all gunfire is never called into 911?.
From a history of working with these data, and many years of familiarity with ShotSpotter and a multitude of other public safety systems, I can make some assertions that I think are important for a better understanding of the technology’s value proposition. First and foremost, I talk frequently about using quantitative metrics to evaluate the success or failure of any new capability or process, and for ShotSpotter, an easy one is their accuracy. ShotSpotter touts a 97% accuracy rate?, and while I would assume that we would all take any number from the company itself with a grain of salt, it is frequently echoed by a loyal customer base. In actuality, the method used to derive this number is inherently flawed. The core of the issue is that ShotSpotter presumes that every alert is real unless told otherwise, but agencies often cannot confirm one way or the other, and are thus reluctant to tell the company that it was wrong. In places where the police are doing their due diligence, the numbers tell a different story, with most pegging the accuracy around 50%?. Again, this isn’t a bad thing, as being 50% accurate on a pool of often unreported shootings is still a notable improvement compared to not responding at all. The simple fact here is that using absurdly high accuracy numbers detracts from the value being provided, as it is easy for opponents to tear down the lofty and imprecise assertions and use that as a cudgel to fight the technology in its entirety.
Similarly, my estimations put the false negative rate, where ShotSpotter fails to activate when it should, to also be around 50%. This is a much easier number to measure and agencies should come to their own conclusions, as I have found the localized rate to be highly impacted by geographical conditions. In any case, no magical dataset is required for this analysis, simply look at an agency’s confirmed shootings within the ShotSpotter coverage area which meet the requirements for activation (i.e., outdoors, unsuppressed, and above a .22 caliber), and determine the proportion of activations to non-activations. I will continue to reiterate, no one should be alarmed by the numbers I present here. The point isn’t to undermine the technology, but rather to put it into perspective so that municipalities, police agencies, and the general public can make better judgements as to the effectiveness of the system, which can translate into less controversy and more support. This perspective adjustment is not only important, it is necessary, as most criticisms are founded on improper measurements of usefulness. For instance, the media makes repeated attempts to judge value from the number of people stopped or arrested based on ShotSpotter alerts, but the knowledge of the shooting in the first place is just as important. These data are immensely valuable in the aggregate, as they inform deployment strategies, but agencies are not connecting the dots between alerts and investigations, at least not in a meaningful way that can be explained to the general public.
All public safety technologies have a notable cost, and it is incumbent on the cities and agencies that deploy such systems to demonstrate their effectiveness with actual data, not anecdotal evidence and subjective appeals. In turn, these data should be publicly available, as there is no real reason to conceal the metrics that determine whether something is useful. Conversely, being secretive drives the assumption that the technology doesn’t work, an idea that gets reinforced when companies and customers provide easily refuted numbers. I’ve been involved in the deployment and operationalization of public safety technologies for nearly two decades, so nothing I’ve researched and written here surprises me, but it shouldn’t have to be a surprise to anyone. These numbers are not a strike against ShotSpotter or the agencies that have deployed their technology, but rather, they represent a meaningful upgrade to the status quo, and an exponential improvement in quality compared to competing providers. The collection of data can build the framework to solve many of today’s challenges, and leveraging that data in a transparent and collaborative way can build trust that permeates society…something so desperately needed in a time when trust in government, and law enforcement in particular, is tragically low.
Footnotes
Dedicated to the support of analysis-driven policing to improve impact, outcomes, morale and improved public safety. SME in crime and crash analysis, operational evaluation and CompStat-driven management.
3 年In other words, to get the biggest bang for the buck, and save the most lives, such applications must be supported at the local level by analysts.