"PHM Doesn't Work"... Says Who?!

Those willing to render a broad brush verdict that “Population Health Management doesn’t work” are falling prey to a common set of logical fallacies that have persisted for the past two decades.[i] Moreover, by posing the matter of evaluation as a dichotomous one, we are undermining our ability to determine what is good, what is better, what is best in this rapidly changing field of endeavor. For the 21st century, it is time we not only got more sophisticated in adapting population health management to the post-reform landscape, but also in modernizing our approach to assessing and assigning value to these approaches so we can accelerate the reliability and impact of these programs.

Fallacy #1: Assumption that PHM programs are comparable, even when they differ in many observable and unobservable ways

When an appropriately powered and rigorously conducted experimental study of a new drug or medical device finds no significant benefit, we feel justified in drawing a tentative conclusion about its effectiveness. These types of interventions are straightforward and static enough to allow standard experimental designs and resulting comparisons. When similar observations are replicated in other robust studies, one can eventually say with confidence that the new intervention “doesn’t work”, at least for the indications and populations studied under the conditions imposed by these experiments. End of debate, at least for those of us who respect the scientific method and inferential statistics.

It’s quite another thing to draw a similar conclusion when a wide variety of differing interventions that happen to fall under a common label (e.g., “PHM” for population health management, or “DM” for disease management) are tested in a range of study designs and evaluated by varying methodologies and may turn out to show a wide variation in outcomes individually. In this case, the respective interventions may have little in common other than sharing the same categorical label, in that they involve different combinations of goals, communications, technologies, incentives, integration with physicians, etc. and are targeted to differing populations. Because they are not comparable to begin with, they cannot be evaluated in this manner.

Fallacy #2: Combining studies of successful and unsuccessful programs and concluding null effect in the aggregate

Related to Fallacy #1, assessing the literature of PHM studies, using widely different approaches to differing populations in different study designs, to conclude that “on average” or “in aggregate”, there appears to be no consistent impact because of the mix of positive and negative outcomes is common.

However, these conclusions are flawed because of lack of comparability and also because enumerating a series of positive and negative studies is not the equivalent of doing a formal meta-analysis that might expand statistical power and supportable conclusions. In most cases, however, the many differences in these interventions and studies would preclude being able to perform a satisfactory meta-analysis.

The relevant issue is not whether everything labeled PHM always works. Rather, the more interesting question is whether any PHM approach ever works – if it does, as demonstrated by rigorous evaluation, then ostensibly we can understand why and figure out how to adapt it to other similar situations, populations, etc. Even if 90 out of 100 PHM programs are were shown to be ineffective, that does not undermine the potential for the remaining 10 programs to be effective if they are substantially different programs. That’s because PHM is not a generic or static intervention – these programs are highly heterogeneous, and are also constantly evolving and innovating to leverage the latest technology, behavior influencers, communication modalities, social communities, available data, etc.

Fallacy #3: Assumption that process measures are valueless in comparison to outcomes measures in evaluating PHM

In the end, we all die – the ultimate outcome measure for defining health status; but finding more proximate and useful intermediate outcomes is necessary for studying PHM impact on health. When these outcomes are determined by multiple influences – both the PHM interventions themselves and extraneous factors as well – then this poses serious attribution problems that interfere with the interpretation of causation. In this situation, adding process measures to provide more intermediate measures that correlate with outcomes and which have fewer attribution problems can be very useful.

Fallacy #4: Assumption that only classical scientific methods are appropriate for evaluating and understanding PHM programs

Some experts, including former CMS Administrator, Don Berwick, MD, have observed that many PHM programs resemble social interventions more than scientific ones, and may need to rely on the different types of assessment used to evaluate social programs, such as realistic evaluation.[ii] While this approach has not been used for PHM often[iii], it should be further explored, as the experimental methods so useful for evaluating new drugs and devices are often not as useful for evaluating innovative social programs, particularly those that are complex and adapting over time.

In a recent Annals of Internal Medicine commentary, Frank Davidoff, MD offered a similar observation in his editorial, “Improvement Interventions are Social Treatments, not Pills.[iv] As he describes, “Improvement interventions, like pills, can change clinical outcomes; unlike pills, however, they do so by applying innovative social treatments that change the way health care is organized and delivered, thereby narrowing the clinical ‘knowledge–performance gap’.” He adds, “The inherent incompatibility between the fixed protocol study methods—seen as essential for knowing whether an intervention “works”—and iterative, experience-based changes to improvement interventions—seen as essential for making them work—clearly presents improvement researchers with an uncomfortable dilemma.” Davidoff suggests that mixed study methods and time series methods may be more appropriate evaluation approaches for these “social treatments”.

Fallacy #5: Assumption that who or what is causing process/outcomes improvements matters more than the process/outcomes improvements themselves

When hospitals or physicians undertake efforts to achieve some of the same impacts that have been demonstrated by PHM programs, rarely is there questioning or criticism of the validity of these results. It’s as if the motives or credentials of the actors creating these outcomes somehow influence the credibility of the outcomes themselves. Needless to say, if a physician or PHM vendor or even a robot can achieve the same improvement in important preventive screening or medication adherence rates, then all of these means deserve the same credit for having achieved the same ends.

Fallacy #6: Assuming PHM programs have not evolved substantially over 20 years since their progenitor, disease management programs, emerged in the early 1990’s

Most current PHM programs are complex, multicomponent combinations of selectively targeted and stratified interventions that may include biometric monitoring devices, online/mobile applications, coach-mediated motivation, incentive programs, social communities and gaming, etc. In many cases, we are seeing providers in value-based, risk-based contracts deploy these interventions on a more local and intimate basis than earlier payer-based interventions that were largely telephonic and remote.

With due respect to my colleague, Archelle Georgiou, MD, the five technologies she named in her blogpost, “The Death of Disease Management (Finally)”, as potential replacements for DM are often included these days in the most progressive approaches to PHM.[v] She is correct that the old DM interventions from the 1990’s and 2000’s implemented most often by vendors on behalf of payers (or the payers themselves) have largely “died” – but they have done so by evolving into more sophisticated, multi-modality programs today that are increasingly driven by provider organizations and delivery systems who have a direct relationship with the populations being managed.

Fallacy #7: Assuming that CMS’s evaluations of PHM programs are the last word on the matter

The variety of different DM, CM, and other PHM programs that were evaluated by CMS in different study designs in the 2000’s cannot be lumped together as though equivalent to a meta-analysis. Having participated in one of CMS’s randomized trials of PHM myself, I recognize the limitations that study design, patient recruitment, confusion about benefits, availability and timing of data, and other factors played in handicapping the outcomes of some of these efforts.

When PHM is performed suboptimally, and unimpressive results are obtained, it is not possible to say whether optimally executed PHM of a similar type would have been more successful. One can only say that a particular CMS trial of PHM, done in the manner specified by CMS and with the limitations and conditions imposed by CMS, did or did not show positive results. In some cases, the PHM Demonstrations did show favorable outcomes on quality and cost; in the majority, they did not. Therefore, we probably should implement these programs in a different manner than CMS did in these demonstrations if there are other reasons to believe they may have merit.

Fallacy #8: Concluding that if better adherence to Evidence-Based Medicine doesn’t solve every quality issue in health care, then it has limited value as a strategy for improving health care value

Despite Dr. Georgiou’s critical comments about EBM in her blogpost[vi], we all recognize the high degree of unwarranted variation in quality across many dimensions of healthcare that have been studied by Wennberg and others, and the gains that EBM approaches have achieved in many circumstances that have been widely publicized. My personal experience of measuring and managing health care value over 30 years convinces me that in many cases, there is more value to be derived from doing better what we know (greater reliability) than by knowing better what to do (greater innovation). In most instances, this is what DM and PHM have been trying to do for the last 20 years, and in many cases they have been successful in reducing unwarranted variation and improving the consistency with which EBM is applied to population health improvement.

Conclusion:

Declaring that “PHM doesn’t work” is somewhat analogous to saying “medical care doesn’t work”. Sometimes it does work as we expect, other times it doesn’t, but since we appreciate there are myriad variables that determine these different outcomes, we avoid such a nonsensical statement. Matching the particular intervention to the particular patients and the right particular time is what medicine is all about. Likewise, we should be focusing on the variables that determine when PHM interventions are most likely to do the most good, and letting observation and analysis of resulting outcomes drive iterative refinement of our PHM models.

This is especially true for today’s complex, multi-component Population Health Management interventions, which Dr. Davidoff might characterize as “social treatments”. Whether we apply realistic evaluation methods, mixed methods, and time-series analysis, we must think beyond the conventions of experimental methods that are unsuitable for these interventions. Another hypothesis to be explored is whether certain interventions that yielded only modest results when deployed by health plans, may be more effective when implemented by providers who can leverage the credibility and trust of the doctor-patient relationship.

It behooves us all to get beyond this level of indiscriminate generalization to begin to define the optimal recipe for achieving the Triple Aim. Neither broad acceptance or rejection of complex interventions is appropriate – we must be agnostic, analytical, curious, unbiased, and discriminating in discovering and improving the recipe for achieving optimal outcomes from a wide array of PHM tools, technologies, and services that have been shown to be effective in many circumstances. We are now too experienced to embrace PHM as a simple panacea or reject it outright because of inconsistent evaluations, and hopefully too wise to throw the baby of PHM potential out with the bathwater of mixed PHM results.