2 Key Ideas To Evaluate Your Quality Improvement System

(Click the Soundcloud box beneath to listen to the podcast version of this article.)

David Kashmer (@David Kashmer)

How would you evaluate a healthcare quality improvement program? Let's say you're looking at your healthcare system's process improvement system and wondering "How good are we at process improvement?" How would you know just how well the quality system was performing?

I've sometimes heard this called "PI-ing the PI", and it makes sense--after all, the idea of building a quality system even extends to learning how well the process improvement (PI) system works.

In the many systems I've either worked in, helped design, or have consulted for I've found the question of "How good are we at PI?" can often be boiled down to a matter of efficiency and effectiveness.

Efficiency

This dimension of the PI process can be thought of as how little waste there is in the PI process. What is the cycle time from issue identification until closure? How much paper & cost does the PI process incur? Do projects take more than 120 days?

The efficiency question is very difficult to answer in healthcare process improvement, and I think that's because our systems are not so well developed yet as to have many benchmarks for how long things should take from identification until closure (for example). I often use three months (90 days) as the median time from issue identification to closure, because there are a few papers that cite that number for formal DMAIC projects.

Now, there are a few important statements here: (1) when I say 90 days to issue closure I mean meaningful closure & (2) if 90 days is a median target...what's the variance of the population?

Let me explain a bit: Lean Six Sigma practitioners are often comfortable with thinking of continuous variables as a distribution with a measure of variance (like range or standard deviation) to indicate just how wide the population at hand is. Quality projects often focus on decreasing the standard deviation to make sure things go better in general. This same approach can be used to "PI the PI" effectiveness. What is the standard deviation of how long it takes to identify and close out an issue for the PI system, for example? How can it be reduced?

These are some of the key questions when it comes to measuring the efficiency of the PI system.

Effectiveness

This dimension is, arguably, more important than efficiency. For example, imagine working really hard to decrease the amount of time it takes someone to throw something away. Yup, imagine working hard on improving how well someone throws away a piece of trash. Making a process efficient, but ultimately ineffective, probably isn't worth your time. (I'm sure there's some counter example that describes a situation where waste disposal efficiency is very important! I just use that example to show how efficiency can be very far removed from effectiveness.)

When it comes to measuring the effectiveness of your PI system, where would you start? Being busy is one thing, but being busy about the right things is likely more important.

One important consideration is issue identification. How does your PI system learn about its issues? Does it just tackle each item that comes up from difficult cases? How do staff highlight issues to PI staff? Is that easy to do? Does your system gather data and decide which issues are a "big enough deal" to move ahead? Does it use a FMEA and COPQ to look at factors that help prioritize issues?

These are some of the most important issue identification factors for your PI system, but by no means are the only ones related to effectiveness.

Once the right issues are acquired in the right way at the right time, where do they go from there? Are all the stakeholders involved in a process to make improvement? Does the system use data and follow those data to decide what really needs to happen, or does it only use its "gut"? Is the PI system politicized, so that data aren't used, aren't important, aren't regarded, or just aren't made?

The staff at the "tip of the sword" (the end of the process that touches patients) and even those who never see a patient but whose efforts impact them (that's every staff member right?) are armed with data they can understand that describe performance. Even better, the staff receive data that they've asked for because the PI/QI process tailor made what data the staff receive. (More on that a little later.)

Once issues are identified, and the PI system performs, what happens with the output? This is another key question regarding effectiveness that can let you know a lot about the health system. There's an element of user design (#UX) in good PI systems. Do the data get to the staff who need to know? Do the staff understand what the data mean? Are the data in a format that allow the data to impact performance? Are the data endpoints (at least some of them) something unique and particular that the staff asked about way-back-when?

Lean Six Sigma is 80% people and 20% math.

You may have heard that old saying. In fact, it's been said about several quality programs. (I've discussed previously that, yes, the system is 80% people but getting the 20% math correct is essential--otherwise the boat won't float!) It is on this point about effectiveness that I'd like to take a second with you before we go:

One of the major items with quality improvement is the ability to use trusted data to impact what we do for patients for the better.

That's the whole point right? If the data don't represent what we do, are the wrong data at the wrong time, or are beautiful but no one can understand them, well, the PI process is not effective.

This, to my mind, is the key question to gauge PI / QI success:

Do we see data impact our behavior on important topics in a timely fashion?

If we do, we have checked many of the boxes regarding efficiency and effectiveness, because, for that to happen, we must have identified key issues, experienced a process that somehow takes those issues and creates meaningful data, taken that data in a format that is understood by the organization, and we must have done it all in a timely fashion that actually changes what we do. That is efficient and effective.

Originally published at TheHealthcareQualityBlog.com and reposted here with permission.