Thoughts from 2017 PDA Quality Metrics Conference

I had the pleasure of spending the last two days networking and discussing quality metrics and quality culture with industry and FDA leadership at PDA's 2017 conference. This is the fourth time this conference has been held and I've had the pleasure to attend the last three, but this one by far was the best of the three. They just seem to keep getting better!

While definitely a smaller crowd than at the 2015 conference, this year's conference included some of the most candid and honest dialog on these topics that I've seen between industry and FDA. So this year, I decided I would share some of the themes and my learnings from the two day event.

Day 1 - Focus on Quality Metrics

The first day focused on quality metrics and specifically the revised draft guidance issued by the FDA in November 2016. Some of the key messages coming from the FDA centered on the goals and benefits of the proposed program as well as their desire for continued feedback from industry. The FDA's metrics program is intended to improve the quality of drugs, accessibility of quality drugs (minimize shortages), and effectiveness of the FDA regulatory oversight. Additionally, the FDA desires to foster a joint culture of dialog with industry focused on what they learn from the data they collect as both sides share the responsibility for providing quality drugs. They know that dialog will not likely come easily at first but hope with time to see improved communication especially around areas that could impact the quality or availability of drugs.

There were many "technical" aspects of the program discussed including:

• overview of the key changes between the first and second drafts including recognition of the valuable feedback FDA received from industry on the first draft
• discussion of who will be doing the reporting and comparison of the benefits of site reporting versus product reporting as the proposed program is geared to collect information in both formats
• phases of the program: Based on industry feedback, there will be a voluntary phase prior to implementation of the mandatory program for drugs and some biologics (see scope information in the draft guidance). There is currently no defined endpoint for the voluntary phase (i.e. I interpret this as we could see several years of voluntary metrics reporting before the program is finalized).
• timing of the program: The voluntary phase will be launch using notification through the Federal Register approximately 1-2 months in advance of portal opening and will include the details on what to submit, how to submit and how long the portal will remain opens. While not mentioned in the guidance, the FDA indicated at the conference that the portal would likely be open for approximately 1-3 months. The FDA anticipates launching the voluntary phase in early 2018.
• metrics and data to be collected: The draft metrics were simplified from first version and focus on collection of data which will be used to determine Lot Acceptance Rate (LAR), Invalidated Out-of-Specification Rate (IOOSR), and Product Quality Complaint Rate (PQCR). These metrics were selected as the agency feels they are indicators of robustness of commercial manufacturing processes (LAR), robustness of lab operations (IOOSR), and voice of the customer (PQCR).
• recognition of participation via the reporters list including discussion of the pros and cons of the proposed tier based approach in the voluntary phase

There were also a couple of interesting questions discussed with the first panel. One of the first questions, as you might expect, was around the potential impact of the new US government administration on this program. I found it interesting that their perspective was aligned with my own. There is always new policy in association with a new administration in the US (either every 4 or 8 years). With that said, the FDA still feels this program is important to achieve many of the objectives set out in FDASIA and the 21st Century Cures Act as well as being a smarter and more innovative way to provide regulatory oversight to industry.

Another interesting question for the FDA panel centered on why quality culture wasn't mentioned in new guidance (in the first draft, it was mentioned nine times). It was clear from listening to the panelists that just because "quality culture" is not written into the current draft of the guidance doesn't mean it's not important. The draft guidance document clarified and focused on a program that would be executable by both industry and the agency. They also feel that the proposed metrics can be very predictive of the future and are driven by an organization's quality culture. Tara Gooen Bizjak, Senior Science Policy Advisory for Pharmaceutical Quality at CDER, used the following graphic to demonstrate the relationship between several of these metrics:

This matrix connects the robustness of an organization's manufacturing and laboratory processes against the organization's lot acceptance rate indicating that those operating in the green quadrant would have more robust manufacturing processes and laboratory operations producing quality product with minimal supply issues. It could also be inferred that those in the green quadrant likely a stronger quality culture.

There was also an entire session devoted to discussion of the analytical approaches from both perspectives. Alex Viehmann, Operations Research Analyst at CDER, presented an overview of the work going on at the agency in preparation to receive and analyze the requested data. Much of the groundwork has been around defining data structures and processes for validating the data from both internal and external sources. They are anticipating a testing phase later this year (likely in the fall) for the submission portal in order to help test the assumptions they've made to date.

Industry representatives also presented their experience in analysis of quality metrics for their own internal programs. Some of the learnings they discussed include:

• acknowledgment that the tools to automate data collection and analysis are out there...organizations do not have to invent something in order to perform robust analytics and reduce the burden on the organization
• metrics programs are inherently complex namely due to inconsistency in definitions and the variety of sources for the data...the complexity of these programs should not be underestimated
• metrics programs should aim to "Find it once, fix it everywhere" especially when there are signals of a larger issue or the potential for an issue to pop up elsewhere in the organization by working on impacting the "drivers of the drivers"
• in a related comment, several speakers emphasized the need to not "sweat the red" when looking at a dashboard but also not to get too comfortable with the green (a great analogy was provided for this: think of a watermelon, the surface is all green and looks good but you never know what lies beneath...it may be good, it may be bad)
• know that whatever you decide to measure changes behavior - sometimes for the better but also potentially for the worse
• don't forget to understand the context around the metric as this can help you determine if a blip is just a blip or indicative of a more serious issue
• don't get overly focused on the tools (e.g. dashboards) but instead focus on making information transparent and flow in an organization
• don't expect your metrics to stay the same year after year...after all, a robust quality metrics program requires continual improvement and will evolve

It was clear that both the FDA and industry understand that benefits of the program may not be realized immediately. It was repeatedly acknowledged that we are at a beginning of a journey. We have a lot to learn in order to achieve the ultimate goals of improving the quality of drugs, accessibility of quality drugs (minimize shortages), and effectiveness of the FDA regulatory oversight (specifically around inspections and post-approval changes).

Day 2 - Focus on Culture

On the second day, we shifted our focus to quality culture. We started with presentations from both the FDA and MHRA discussing their perspective on quality culture. From the FDA, we heard about culture as both a noun and a verb which begs an interesting question....are you looking at culture as a thing (something you possess) or a behavior (something you can grow and influence)? And if culture is more of a behavior, then we were reminded of the way adults learn behaviors....via the ladder of inference.

The perspective from the MHRA indicates quality culture requires knowledge, diligence, vigilance, management commitment, and transparency. Based on their 2015 inspection experience, many of the serious failures resulted from an absent or over-controlling senior management or a lack of vigilance. In fact, they still find that many of the conclusions from the Clothier Report (MH 149/1794) issued in 1972 still hold true and reminded us that people are still at the center of what we do.

"[there are] no technological advances which eliminate the need for skillful personnel devoted to their work" ~ Clothier Report (MH 149/1794)

When industry presented their perspectives on quality culture, I must admit I was quite pleasantly surprised with what I heard from industry's senior leaders. There was a significant portion of the time spent on the need to return to W. Edwards Deming's 14 Points for Management (first presented in his book Out of the Crisis) and in particular, point number 8 - Drive out fear in order to build and sustain healthy and robust quality cultures....and I couldn't agree more.

A company with a highly developed culture of quality spends, on average, $350 million less annually fixing mistakes than a company with a poorly developed one." ~ Harvard Business Review

Then we heard about the progress that the PDA team is making on their Quality Culture assessment tool including perspectives share by one of the participants. The tool provides a structured framework aimed at assessing the quality culture of an organization. In the pilot of the tool, there are approximately 50 sites from 26 firms participating and 64 assessors have been trained. PDA is targeting finishing the pilot this year and rolling out the tool formally soon after that.

Machelle Eppler, Vice President and Head of Global Quality Compliance & Regulatory at Patheon, shared their experience so far in using the tool (five sites from several regions of the world participated). She re-emphasized that the tool provides consistent language, framework and scoring method as well as a road-map for improvement. She also discussed the need to create the right environment (make people feel safe to be open and honest; there are not right and wrong answers) and to spend the necessary time planning (several weeks not days). They also found senior leadership engagement is the key...in fact, all communications around the process came from plant GMs and not quality. They also felt it important to communicate results and action plans - share successes and implement best practices. After all, she reminded us that improving quality culture is a journey.

We ended the conference with one final panel discussion where we were reminded of the goals of the program we had discussed the previous day:

• improving the quality of drugs
• decreasing drug shortages by detecting signals earlier
• decrease inspection frequency and/or duration
• increase dialog between industry and FDA

As I'm sure you can understand, there was a lot of information shared over the two days and it would difficult to capture everything we discussed in one summary. If you're interested in more information about the conference, help in improving your organizational quality culture or questions on getting ready for the FDA’s quality metrics program, please contact me at [email protected].