What is Delivery-driven Policy?

Why is policy still educated guesswork with a feedback loop measured in years? 

—Tom Loosemore, Co-Founder of the UK’s Government Digital Service

(The full paper on this topic lives here.)

Policymaking is in a quiet crisis. While there are meaningful differences of opinion in the United States and elsewhere about which policies will result in the kind of society we want (often driven by different visions for our society), it’s also true that many of the policies that result from our democratic processes simply do not live up to their intent. It’s like a car where the steering wheel is only loosely connected to the wheels: We might fight over who is in the driver’s seat, but regardless of who is driving, you’re not going to get where you meant to go — and you’re going to hurt people along the way.

Why is there such a profound disconnect? To be clear, implementation is often derailed from its intent by the deliberate interference of people who oppose the policy. But much of the failure is not deliberate; it is the result, in part, of an outdated model that keeps policymaking and policy implementation as separate domains, with separate skills and incentives. The impact of these outdated models becomes increasingly dire in a shifting landscape where the work of government grows more and more complex.

How might the shift to a digital world affect government’s ability to implement policy? In theory, government could use digital technology to build programs with the feedback loops needed to course-correct over time, effectively steering the car. Some have suggested we just need to invest more in technology for government; but it’s not how much government spends, it’s how government spends it. Constrained by antiquated procurement practices, and focused on outputs rather than outcomes, the US spends $200 billion a year on digital technology but frequently lacks the data and insights needed to effectively manage the programs it funds. And it is particularly hard when measuring those outcomes requires crossing silos, as our government was designed around discrete functions in a pre-internet era.

But Code for America, the United States Digital Service, 18F, and many others have made the case that a user-centered, iterative, and data-driven approach can result in digital technology that provides those much needed data and insights, and at a far lower cost. The real benefit, however, is when those same practices — user-centered, iterative, and data-driven — are applied to the policymaking process as well.

So what is delivery-driven policy?

Today, emerging models of policymaking and delivery, both theorized and increasingly practiced, are resulting in better fidelity to the intentions — reconnecting the steering wheel to the wheels of the car.

These models involve integrating policymaking more tightly with its implementation, and iterating on both in consistent cycles. They serve in stark contrast with traditional “waterfall” models, in which policy development is a distinct project phase managed by a distinct team with distinct skills, and implementation begins with a separate team with separate skills after policy is set. The traditional model looks something like this (borrowed from former Code for America Product Manager Jake Solomon’s 2015 talk at Code for America Summit):

By contrast, a delivery-driven policy process assumes policymakers will get things wrong the first time. It seeks to be agile, iterative, and user-centered, with how the policy will be delivered informing its development from the start. This requires multi-disciplinary teams that include digital and design professionals, with the skills to accurately understand and solve for user needs, alongside policymakers, subject matter experts, and other stakeholders like procurement and compliance professionals.

In this model, teams start small and iterate progressively on both the policy and delivery together. They also build instrumented delivery systems that provide near-real-time feedback, creating mechanisms for experimentation that can inform policy development with the knowledge of what’s actually working towards the original intent.

Case Study: Medicare Regulators and Real-Time User Testing

I’m realizing that what I’m holding is 500 pages of untested assumptions!

—Tom Loosemore

When federal regulators are charged with implementing a law, common practice is for the regulators to spend months detailing the rules that will govern implementation. Often, a tech team (usually an outside vendor) will be charged with encoding those rules in the form of a website that users will interact with. Traditional development methods call for “user acceptance testing” at the end of the project, which is generally too late for meaningful feedback. Agile development, which insists on user feedback throughout the process, is becoming more accepted, increasing the quality and decreasing the cost of projects in the public sector. But while the tech teams may be able to learn from users, they can be very constrained in how to respond by the parameters set out by the regulators, who rarely see how what they’ve written plays out in the real world. 

In 2016, regulators at the US Department of Health and Human Services (HHS) were charged with implementing the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA), which was meant to enable Medicare to incentivize better care via changes to the payment system for physicians who treat Medicare patients. HHS tapped a team from the United States Digital Service led by Mina Hsiang. Mina and her team proposed to do things differently. 

Instead of waiting months for the final regulations to be delivered, Mina asked for a first draft and had her team build early prototypes of website features based on the draft. They tested these prototypes with real users, and shared data and feedback with the regulators. In testing a “setup wizard” assistant for doctors, for instance, they found that the criteria for enrollment were not actually mutually exclusive and collectively exhaustive (see the MECE principle), which meant that doctors could fall into multiple buckets. This was not only extremely confusing for doctors trying to use the service, but it led to users who were not properly enrolled. This problem had not been visible to the policy team, but because Mina’s team was able to surface it in an early implementation, they were able to engage the policymakers in changing in the language of the rule (and the tools), and also changing the criteria (when the rule was finalized). 

Needless to say, in a traditional policy and delivery team relationship that did not involve real user feedback during the building of the service, these problems would have become apparent when the service was already operating at scale, compounding problems for doctors and administrators alike. It’s also likely that the vendor, having delivered to the specification they’d been handed, would have been moved into a maintenance contract, making it hard to quickly make the changes that were needed. In a waterfall approach, delivering on the specification is the end of the work. In an agile, delivery-driven approach, it is the beginning of figuring out how to make the service really work for its users.

Another way the delivery and policy teams worked together was around how doctors received data and information about their performance. Remember, the point of MACRA was to incentivize doctors to provide better care for Medicare patients by paying them more when patients reported better outcomes and better experiences, when care was better coordinated, and when services were used appropriately, according to the measures the program designed. This meant that it was critical that doctors understood when and how often they’d met those criteria, so the system was supposed to collect data from doctors and provide analysis back to them. 

In this case, Mina’s team looked at how doctors used the existing services from the programs that MACRA was intended to replace. They found that doctors were asked to upload their data in ways they found enormously confusing (and were often left wondering if the submission had even gone through) and then a year later they were invited to download a PDF report containing a series of charts which they didn't really find useful. These previous systems had been built at enormous cost, for the exact purpose of providing this analysis to doctors, but only about 5% of doctors were even using it. (If you divide the cost of the feature by the number of doctors pulling their reports, CMS was spending about $1M per report.) Armed with this insight, Mina’s team was able to work with policymakers not just to improve on the upload features and change the reports, but to change the process by which they would be designed. Policymakers decided to hold off on defining what the report should contain in the final rule, and Mina’s team instead helped them outline a user research-driven process that would take place later in the year which would iteratively determine what data doctors could get back and how it would be presented. The result was a report that doctors actually use.

Even the name of the program (not generally the domain of the tech team) benefitted from user testing. Many of the proposed names were confusing to the doctors, but it was important that this sounded like something they could understand and trust. The delivery team took just two days to talk to 50 doctors using a questionnaire and brought the data back to the team. This research led to calling it the Quality Payment Program (QPP).

The website and the regulations went through many more iterations before the QPP team called the rules final. The results of these collaborations were happy policymakers, happy doctors, and a program that is succeeding in its goals. Because policymakers could see how users experienced and interpreted the rules they’d drafted, and make changes to make them work better, they were proud of their work. Many of them reported that they’d just written the best rules of their career, and Medicare patients are now getting better care.

There's more!

A longer paper with more examples and case studies, can be found here.

In collaboration with New America and the Delivering Better Outcomes Working Group hosted by the Beeck Center for Social Impact and Innovation at Georgetown University, with whom we’ve developed these ideas, we invite all readers to share their own examples of delivery-driven policy at [email protected]. In subsequent papers, we will go deeper into some of the examples introduced here, highlight new examples submitted by our wider community, and expand on these ideas as they relate to adjacent fields—including advocacy, procurement, and digital talent in government. You’ll find these on the delivery-driven government page of Code for America’s website.