Mythical Medical and Little-Known Knowns: Metrics for Notified Bodies, Manufacturers, and Regulators
David Rutledge
CEO & Founder of Global Strategic Solutions, LLC | PRRC for the Medical Device Sector | Best Selling Author, Mythical Medical | Podcast - What's in Your Strategy? | Lead Auditor ISO 13485 | Notified Body - BSI Trainer
Notified Bodies, manufacturers, and regulators have leaders. Communication among these entities can be tough and complicated. Great leaders within these organizations sometimes only desire three things: (i) predictability, (ii) approvals on time, and (iii) quality submissions. These represent shared common aspirations that provide a platform for mutual collaboration.
Yet, there must be a way to structure a program to facilitate communications using metrics. At its heart will be the need for agreed-upon data. Product submission metrics did not exist at Mythical Medical. People should be brought together to take a deep dive into these processes and then agree on the future, immediate next steps.
Ideas Are Needed
Ideas are the natural-born enemy of the way things are. Kai-Lin, an employee at Mythical Medical, is unaware she is on a journey to greatness (see previous articles). What she is keenly aware of is that serious communication challenges exist between Mythical Medical and regulators as well as their Notified Body. She and her team are in unfamiliar territory. Improvement in working relationships with their Notified Body will take priority now due, in part, to the increased number of planned product submissions.
In this fictional story, I’ll guide Kai-Lin through a process to generate ideas that can be used to create and implement a communication strategy. I will be addressing relevant topics and providing simple, practical solutions. These principles can be implemented globally with regulatory agencies, too.
A Benchmark Is Needed
Previously we learned Kai-Lin was all about making decisions based on evidence. She demonstrated that by using surveys to acquire specific sufficient clinical evidence (safety and performance data, in this case, to fill in a clinical evidence gap) and later overseeing the creation of a framework to institute a benefit-to-risk process.
Notified Body Designations and Expectations
Check the NANDO (New Approach Notified and Designated Organisations) Information System website. The Member States, European Free Trade Association (EFTA) counties,
EFTA countries (EEA members) and other countries with which the European Commission (EC) has concluded Mutual Recognition Agreements (MRAs) and Protocols to the Europe Agreements on Conformity Assessment and Acceptance of Industrial Products (PECAs) have designated Notified Bodies, established per directive and regulation. Lists of Notified Bodies can be searched on the NANDO web site. The lists include the identification number of each notified body as well as the tasks for which it has been notified, and are subject to regular update.
Kai-Lin's reputation continues to grow with regulators and Notified Bodies all over the world. She and her team are respected. However now, she is lacking data again.
During our first meeting, I asked Kai-Lin, “How are discussions going between Mythical Medical and your Notified Body?” She replied, “Some describe it as challenging.” We discussed why that was, and it became clear that pieces of information, data that could be available to assist with communication were not known. Without data, there was no benchmark to measure it against. Without a benchmark, key performance indicators (KPIs) could not be developed. Thus, the first initiative or proposal would be to gather data using metrics, data that had to be agreed upon by her team, and their Notified Body. Kai-Lin, like many others, needed to discover those little-known knowns.
Agreeing on the Data to Collect for Post-Market Clinical Follow-up (PMCF)
Most of the world make decisions by either guessing or using their gut. Having data to establish an initial benchmark is not only a useful tool for communication, but it is also a competitive advantage. The data collected initially will change or shift over time. Some call this data swirl. Carve out time to bring important stakeholders together for a robust discussion.
Agree on what the current pain-points are. Use this information, in part, to agree on the final data to collect. A key here is to filter out ideas of what could be measured to what should be measured. Although this will take some time and a little more effort, the desire to gain efficient product approvals will numb the pain it takes to get you there.
1. Little-Known Knowns: Predictability
Mythical Medical should work with their Notified Body on agreed-upon metrics related to predictability. Here are four metrics to better understand factors that affect predictability: (i) Number of pre-notifications provided versus the total numbers of submissions (percentage), (ii) Percentage of pre-notifications provided shorter than eight weeks before submission, (iii) Number of project cancellations versus total projects (percentage), and (iv) Number of postponed projects versus total projects (percentage). The first two metrics imply that early communication is essential. It can make or break a medical device's successful journey to the market.
Depending on the software or analytical capabilities of the Notified Body, these four metrics could be modified, or others could be added. The point is having this type of information can be useful both to the Notified Body and the manufacturer. It makes for easy graphing, and it can be tracked over time.
2. Little-Known Knowns: Approval Timelines
In business, approvals coming ahead of or later than schedule cause both upstream and downstream manufacturing challenges for people, processes, production, and eventual implementation of launch strategies. Thus, the best-case scenario is to plan thoroughly and have your product approvals hit the targeted date.
These three metrics should be tracked: (i) Number of delayed projects (not postponed projects), (ii) Within project milestone dates, e.g., submission date, 1st Notified Body review round, 1st responses by the manufacturer, 2nd Notified Body review round, 2nd responses by the manufacturer, number of additional review rounds, certificate date related to review closing data, and (iii) Overall project timelines or planned versus actual achieved date.
Data within these little-known knowns can be used to understand better and then characterize specific issues related to submission processes at both organizations. It is okay that you might find embarrassing or frustrating elements. Those are better known than unknown.
Project managers will find information-rich data here. Evidence from these metrics should be segmented to analyze details within individual projects. Find trends in what works and what doesn’t work. Action items can be developed that will have positive effects on all future submissions.
3. Little-Known Knowns: Quality
If you are struggling with quality metrics during the submission and approval process with your Notified Body, here are five simple metrics to start with: (i) Number of questions per pre-notification, (ii) Number of screening deficiencies raised during administrative quality checks, (iii) Number (average/median) of review rounds per project, (iv) Number (average/median) of deficiencies per project or review module, (v) Number of post-approval activities, follow-up items, and certificate limitations.
You get what you measure. What each organization finds will eventually change who they will become in the future. These metrics will create an initial understanding of the quality of your submissions. Otherwise, how would you know where challenges exist and who should be accountable? A nice outcome here is that you can update your submission checklist and train employees to minimize these delays. You can look at these metrics per project team or product class.
Closing Thoughts
When communicating with Notified Bodies, manufacturers, and regulators, the single biggest conundrum is the illusion that communication has taken place. If you want to examine this conundrum, you must have metrics like these. Without them, your competitors will eat your mistakes. Therefore, it’s time to discover those little-known knowns.
Become a great leader within your organization and implement metrics that address: (i) predictability, (ii) approvals on time, and (iii) quality submissions. If you and your Notified Body are struggling to create these metrics, work together to agree to begin somewhere. Metrics will evolve or even swirl over time. That is the natural evolution as data matures or is analyzed. With this evidence, benchmarks can be created. Then, KPIs can be developed to report on progress against the desired result.
This approach seems simple on the surface. However, to manage Kai-Lin’s expectations, I jokingly informed her, “Half the people will love the data, half will hate the data, and half won’t understand it.” However, in time, jewels of truth and pearls of understanding will emerge. Someone has said, “Listen to the experienced people in your life--not because they're always right, but because they have a better understanding of being wrong.”
To pursue excellence, it takes relentless effort. If you want to go beyond what's expected, you must evolve and grow too. Little did she know (and those like her), that implementing these metrics would take her down a path to become a fantastic leader. She is on a journey to greatness.
Metrics like these are relatively new for most organizations, but with changes in product approval and post-approval regulations around the world, deadlines are rapidly approaching. It’s time to collect this data and have discussions about future submissions. Some will say it’s too early until, unfortunately, it’s suddenly too late.
The author would like to thank Bassil Akra formerly of TüV SüD.
David R Rutledge, Pharm.D., FCCP, FAHA, President & CEO, Global Strategic Solutions, LLC, Silicon Valley in California. +1 (630) 846-0350 cell.
Disclaimer
This is just a bit of fun. The purpose of articles like these is for the free exchange of questions, ideas, or comments regarding issues of global interest involving medical regulations and is not to be considered legal advice. These are just opinions and not advice; opinions may change over time, too. Use it at your own risk. This disclaimer will occasionally be posted or updated throughout the year.