At The Intersection Of Quality And Metrics: What's Ahead In FDA’s Effort To More Objectively Measure Quality

US FDA is banking on developing quality metrics to not only aid in selecting which device facilities to inspect next, but to also raise firms' overall approach to quality above baseline. Metrics developed by the Medical Device Innovation Consortium (MDIC) is one piece of the puzzle – but the agency also wants its own set of metrics that it hopes will drill down even further to ensure best quality practices.

"When you look at quality metrics, the areas that we're focusing on are basically how firms structure quality metrics, the kind of metrics we should consider, and which metrics have the most utility for FDA, as well as for firms. Simplicity versus relevance to quality," said Robin Newman, director of the Office of Compliance within FDA's Center for Devices and Radiological Health.

Quality metrics are measures used to assess the overall quality of medical device manufacturing.

"There are a lot of times when something is a very simple metric. Every company knows, for instance, how many open CAPAs [corrective and preventive actions] they have. At least I would hope so," she said. "But that's a very simple metric. Does it even mean anything? Not likely."

"If you look at your CAPA system and you find out that 30 percent of your open CAPAs are open because they failed to meet effectiveness requirements, that might say something about the integrity and the robustness of your quality management system," CDRH compliance chief Robin Newman says.

However, "if we look at that metric and maybe refine it a little bit and say, 'How many CAPAs do we have that are open because of a failure to meet completion requirements or a failure to meet effectiveness requirements?' then that's a very different metric, and that's a meaningful metric," Newman said.

"If you look at your CAPA system and you find out that 30 percent of your open CAPAs are open because they failed to meet effectiveness requirements, that might say something about the integrity and the robustness of your quality management system," she added.

But FDA wants to be sure that a quality measure such as the one imagined by Newman is indeed significant. That's why the agency is in the beginning throes of a pilot project to determine which quality metrics are top-notch – and FDA plans to begin its work by piggybacking off of quality metrics efforts by MDIC.

MDIC recently wrapped a pilot project using three distinct metrics it devised to finger gold-star quality practices that can be used across an array of manufacturing organizations. (See the section below: "MDIC's Quality Metrics Explained.")

Quality metrics are an important issue for FDA; CDRH included development of metrics in its two-year strategic priorities list issued in January. The calendar-year 2016-2017 list directs the device center, by the end of September, to "develop metrics, successful industry practices, standards, and tools that manufacturers can use to evaluate product and manufacturing quality beyond compliance with regulatory requirements."

For FDA's metrics gathering, "We are asking ourselves, 'Why not make use of what MDIC has already done with their quality metrics?" said Bill MacFarland, director of the Division of Manufacturing and Quality within the FDA device center's Office of Compliance.

MDIC's metrics "have already been developed, lessons have been learned, and it’s an opportunity for FDA to use their approach," MacFarland noted.

But the agency is looking beyond MDIC's three key metrics. "We foresee developing some kind of device-specific metrics to be used with those quality system metrics that [MDIC] has developed. And you might be saying, 'Oh, no. Here we go. FDA is introducing some unknown device-specific concept, and that’s going to be where the real challenge is,'" he said.

But MacFarland said firms have nothing to worry about. He points out that they will have a leg-up thanks to eight forthcoming product-specific "critical-to-quality" guidance documents coming out of the device center as part of its larger "Case for Quality" initiative. (Also see "Next Step For FDA's 'Case For Quality'? 8 New Guidance Docs" - Medtech Insight, 3 Nov, 2015.) The first two CtQ documents will cover catheter coatings and abdominal surgical mesh; the other six product types have yet to be identified.

"Since that information [for the CtQ documents] came from a lot of stakeholder engagement, why not use a subset from one of those CtQ guidance documents to identify device-specific metrics?" MacFarland said.

Further, "we’re discussing the potential for two other metrics, one regarding product availability," he said. "This relates to something we run into occasionally in FDA where there's some kind of [device] shortage, and we’re just thinking, 'Wouldn’t it be nice to have some kind of metric information on firm approaches to preventing shortages, ensuring the availability of the products?' That’s still something we have yet to discuss."

"We're very sure we would utilize the MDIC quality system metrics and then some device-specific metrics. We’re still discussing the potential for culture and product availability metrics, as well," FDA's Bill MacFarland says.

MacFarland noted that CDRH Director Jeff Shuren has also expressed interest in developing a metric built around company culture. "We’re just starting to discuss what could that be, and whether that would be something we would want to introduce in the pilot that we will launch this year," he said.

"So, to reiterate, we're very sure we would utilize the MDIC quality system metrics and then some device-specific metrics. We’re still discussing the potential for culture and product availability metrics, as well."

Brian Motter, worldwide VP of quality for Johnson & Johnson's diabetes unit, is looking forward to seeing what type of cultural metric would be developed, noting that it would be a helpful tool when selecting vendors.

"It would be interesting to see how [FDA is] going metricize culture, but when I walk into a supplier, when I was a supplier quality guy, or when I was in operations trying to evaluate if this firm is going to be a good supplier for me, I could walk in right away and tell if the culture was a quality culture," Motter said.

Further, "I could look at how [the supplier is] looking at their lagging indicators, and see the actions that they’re taking, and seeing if they’ve got a mature quality system," he added.

And before the September deadline is up, the agency needs to decide on a weighted scoring system for use with its quality metrics. FDA must ask itself: "How do we come up with a scoring methodology that recognizes that firms have different approaches to quality control, have different specifications and have different levels of assurance?" MacFarland said.

FDA also wants to tie its quality measurements to another Case for Quality initiative: the so-called "maturity model."

FDA uses a conventional maturity model for recognizing the stages that firms go through in their approach to quality – anywhere from early-on firefighting, to a more mature posture where the company makes sure that quality is part of its business so that customer needs are met. (Also see "Will ‘Maturity Model’ Pilot Mean Fewer US FDA Inspections for Medtech? " - Medtech Insight, 24 May, 2016.)

"Quality metrics and a maturity model – they need to interface somehow," MacFarland said.

MacFarland and Motter spoke at MDIC's June 28 Case for Quality Forum, while Newman's comments came May 4 at MedCon 2016.

FDA Pilot Targets Manufacturers

Next, by Dec. 31, FDA will launch a voluntary metrics pilot by selecting 10-20 device manufacturers that will submit quality data to the agency. FDA will then test the data against its quality metrics with a deadline of March 2017. Participating firms found to have above-the-board quality programs will fall off FDA's 2018 risk-based work plan, which informs the agency determinations of which manufacturers to inspect.

"Firms that come out at the highest level of our aggregate scoring system … wouldn't be on the list of firms that will get an inspection" in 2018, MacFarland said. "Firms that are somewhat above the quality floor will be subject to an inspection, but it will be an abbreviated Level 1 inspection.

"For other firms that don't submit quality information to FDA under the pilot – and for those in the pilot that don't meet the quality floor – they will receive a comprehensive Level 2 inspection in 2018," he added.

But skipping FDA inspections might not be what industry is necessarily clamoring for, says Marla Phillips, director of Xavier Health at Xavier University in Cincinnati and co-chair of MDIC's quality metrics working group.

MDIC asked industry, "'What would you hope comes out of the CDRH metrics initiative?' The answer is probably not what you would expect, such as, 'Well, since I'm low risk, FDA wouldn't inspect me as often. That would be great,'" Phillips said.

Instead, according to Phillips, the firms said, "We don't want inspections less frequently. We value the dialogue with investigators, and if FDA comes once every five years instead of two, and they have a finding, it could implicate five years' worth of work." Phillips spoke June 27 at the Association of Food and Drug Officials (AFDO) annual conference in Pittsburgh.

"What they do want is fewer inspections per year. Right now, they have FDA, the EU, Japan, Australia and Health Canada all coming in a given year," Phillips said, noting that the Medical Device Single Audit Program, administered by theInternational Medical Device Regulators Forum, should help to alleviate duplicate inspections.

"I’ve heard different opinions from different people whether reduced inspections is an incentive. Some say yes. Some say you can’t remove inspection all together," Stryker's Chris Hoag says.

Nevertheless, Phillips said, manufacturers believe FDA's use of quality metrics will help level the playing field when it comes to inspections. "Firms will say, 'I'm constantly targeted by the agency for inspections. What about my competitors? Why aren't they being inspected?'"

Chris Hoag, director of global CAPA and quality eSystems for Stryker Corp., agrees with Phillips that fewer agency inspections isn't the only perk firms are looking for.

"I’ve heard different opinions from different people whether reduced inspections is an incentive. Some say yes. Some say you can’t remove inspection all together," said Hoag, whose company took part in MDIC's pilot metrics program.

Rather, it's being publicly labeled as one of the best manufacturers when it comes to quality control that many firms strive for. "When I think of something like that, I think of Fortune’s Top 100 companies to work for as an example. It’s voluntary, you don’t have to participate, but for companies that submit their information, this is a perk," he said at the MDIC forum.

"That’s something that could be an outcome [of the FDA pilot] that might be a potential value-add for companies," he said. Further, "what if there were incentives that reduced regulatory burden, a faster 510(k) turnaround time, fewer oversights or less regulation to actually get product to market – something like that.

"Because if you’re demonstrating higher levels of maturity or better performance in quality metrics, then those might be some real benefits that would drive companies to want to be a part of the FDA pilot program."

Manufacturers can also use the metrics to determine how they should spend their own limited resources, says Patrick Caines, director of quality & post-market surveillance at Baxter International Inc.

"The bottom line is, if FDA agrees to the premise of quality metrics and the supporting data being piloted, then we do see some value, of course, in terms of potentially fewer inspections, but what are the benefits to firms beyond that?" Caines said March 15 at the Medical Device Quality Congress in Rockville, Md.

"Well, I think the metrics will demonstrate that firms that achieve gold and silver levels of quality will achieve and sustain an overall higher quality, allowing that firm to expand their resources in a more targeted way," he said. "So that's the value driver for the leadership of your companies."

Concerns About Timing

FDA's MacFarland is concerned about the timing of the pilot. "One thing that's really on my mind is how to … launch a pilot through a Federal Register notice and give firms enough time to gather information," he said. "We need to launch the pilot by December of this year and get firms' [quality] information by March 2017, so I'm trying … to give firms as much time as they need to gather that."

After the data is collected, FDA must decide how the information will be analyzed. "Firms have different approaches because they choose to. How [can we] evaluate that to made a decision on how FDA will use its inspectional resources?"

Finally, by December 2017, the device center must "propose a voluntary program to recognize independent evaluation of product and manufacturing quality," according to the strategic priorities list. That is, the agency needs to determine how third parties, such as notified bodies, will use FDA's metrics when inspecting.

"FDA specifically and deliberately made clear that we were going to look for any measures or metrics that would be indicative beyond the compliance threshold, above the compliance baseline," FDA's Kristin McNamara says.

"We want to make sure that whatever we propose for a third-party approach, it has utility for FDA and stakeholders," MacFarland said. "That might be a reason why some past efforts on third parties haven’t been highly utilized – because they didn't have utility for device firms."

Such "past efforts" include attempts to offer third-party inspections – such as the Pilot Multi-Purpose Audit Program (PMAP) and the Accredited Persons (AP) program – which both failed due to a lack of industry buy-in.

Why Metrics?

A year or so before metrics were included in the 2016-2017 strategic priorities list, FDA partnered with Xavier University to identify key metrics for industry to assess their own product-quality risks.

In May 2014, "FDA specifically and deliberately made clear that we were going to look for any measures or metrics that would be indicative beyond the compliance threshold, above the compliance baseline," Kristin McNamara, senior advisor to the FDA deputy associate commissioner, said at the MDIC forum.

"We wanted to look deliberately and specifically without any preconceived notions of how FDA might one day use the metrics, and to invite some of the most involved folks in from a range of companies, small and large, to talk about the use of metrics internally and what the best practices are about using them," said McNamara, who co-chairs MDIC's quality metrics working group with Phillips.

"The idea was to have industry and firms own their own quality," McNamara said. "So it was not going to be anything that was going to be used necessarily for a company-to-company comparison, but rather it would be an internal signal against which you are telling us you take action, and how that action is indicative of quality."

"Metrics are just numbers. The numbers don't make decisions for you. It's a number that you want to use to inform your decisions. That is the key to this: not looking for what the number should be," Xavier Health's Marla Phillips says.

Phillips offered her own recollections. "When we started this project, [former CDRH compliance head Steve Silverman] said, 'I have no idea how or if FDA will use your work. Your goal is to come up with something that will be meaningful for industry, which is really in line with the Case for Quality priority, where we want to shift that ownership to industry," she said.

At the time, "I underlined the word 'measures,' because I didn't understand, 'Why are you distinguishing measures and metrics? What is the difference?'" Phillips said.

"But FDA specifically asked to identify measures which could be converted into metrics, meaning, 'This is what we want to measure. Here's the numerator, here's the denominator.' So the agency wanted us to find key measures that we could look at across the risk-to-product quality profile," said Philips, who noted that the device-side metrics were created in tandem with quality metrics for the drug arena.

After a year of working on the metrics, FDA and Xavier handed the torch to MDIC, which adopted the agency's Case for Quality initiatives at the same time.

"When they did that, MDIC said, 'OK, we like to have yearlong projects,' so when we presented our findings, we had 17 measures because we did it across the total product lifecycle. And MDIC said, 'Pick three of those measures and convert those into metrics and run a pilot study.' So, that's what we did from May of 2015 until June of this year," Phillips said.

MDIC's Quality Metrics Explained

In the end, MDIC's goal was to arm industry with practical metrics so a "right-the-first-time" mentality was shifted as close as possible to the initial days of device development.

The group's system of metrics is intended to inform company decisions and trigger action when necessary. The three metrics that were developed address device pre-production, production and post-production actions. They also address industry consideration of how to implement enterprise-wide continual improvement.

Xavier's Phillips reminds manufacturers: "Metrics are just numbers. The numbers don't make decisions for you. It's a number that you want to use to inform your decisions. That is the key to this: not looking for what the number should be."

Each of the three metrics will be accompanied by a best-practices document – currently under development – that will help firms understand how to use metric outputs to inform decisions and prompt particular actions, as well as to aid manufacturers in understanding how to calculate each metric.

Below is a synopsis of how each of the three metrics will work:

The pre-production metric tracks the number of changes that occurred during the transfer stage that were triggered by product and/or process inadequacies. This metric helps to signal the frequency and volume of changes that could possibly have been avoided by a more robust research and development (R&D) system.

By tracking the metric, the firm has information that can inform the decisions of senior leaders related to potential improvements to the R&D process. For example, upon review, it might be recognized that the rigor of the voice of the customer could be improved, that there could be a more thorough evaluation of literature, or even that improved human-factors studies are needed.

"The pre-production area was probably the toughest area we had to develop a metric for," Phillips said. "We kept hearing the comment, 'We're R&D. We're supposed to be doing trial-and-error. How can I measure how many changes I had to make? We're supposed to be making changes, and improving and testing.'"

The answer to that, according to Phillips, "is at some point you're saying to yourself, 'I've got it. Now I'm ready to transfer it.' From that point, how many changes are you making? Or can we measure how effective you were from there? Once we started talking like that, firms in our pilot were able to identify some things that they could measure themselves," she said.

For this metric, it's up to the manufacturer to define "project." For example, does the firm want "project" to indicate the total finished product, or does it want "project" to be the last step in its pre-production process?

"If you then take what you know about each individual project, you can look across your total number of projects," Phillips explained at the June 28 MDIC forum. "How many changes are you having, again, based on product and process, across the total number of projects coming up?

"Then you can see, 'OK, from the R&D group that's feeding products to me, how many times are we getting projects from them that require a lot of changing? So it gives you an idea of the R&D group's effectiveness – not just product," she said.

"That's important because if you see a systemic trend, then you can go back and look at how you are gathering development data. Firms can ask, 'How can we make it more rigorous so we don't have these changes?' But again, this has to be commensurate with the needs," Phillips added.

"If you're seeing that you have three changes, or two changes and they're minor, don't drive a huge, company-wide initiative to drive that down and pull resources from things that are actually needed. But you'll know. You'll know when it's too much for your company, and when it's causing problems and costing a lot of money."

At the June 27 AFDO meeting, Phillips reiterated that one of the pre-production metric's goals "is to help senior management see what's going on, and unless you measure that or have a way to get that bubbled up, it's difficult. You just know people are complaining and that there's a lot of churn, but you just can't put your finger on it. The pre-production metric will help."

Baxter's Caines says top management's interest in outcomes drove metrics activities at his firm while it was part of MDIC's pilot.

"Executives really see the value, not only from a risk-based inspection approach from the agency, but I think also in looking at where our quality stands relative to our competitors, and also getting some insight in terms of the investments we’re making in improving these products – are they really making a difference in the marketplace?" Caines said at the MDIC forum.

"Giving that benchmarking perspective is a big plus for this initiative because that kind of data is awfully hard to get, and it often lacks the granularity that this initiative can potentially offer," he said.

The production metric is the "right-the-first-time" measurement that many manufacturers already track. MDIC recommends triaging the root causes such that resources – employees, capital, etc. – can be focused on areas that will result in true improvement, and therefore, a reduction in risk to product quality.

Additionally, MDIC suggests that firms use root cause trend analysis in such a way that any nonconformances related to product and/or process inadequacies are relayed back to R&D through senior management. Again, this type of review will enable the organization to assess the effectiveness of its systems and processes.

"The strengths of this metric are tracking right-the-first-time based on product and process inadequacies," Phillips said. "We can track and trend within and across lots on a rolling basis to identify the highest area of risk. You can apply predetermined action limits. And again, we said we want to inform decisions and trigger action, so is the number good? I don't know. Maybe a low number for one product is actually good. It might even be world-class. So, you have to decide based on your product profile risk what your action limits and trigger limits are.

"The metric is not skewed by volume. However, the volume in this particular case gives you some insightful information, so it is good to know the number of units started because it's very different to say you have 50 right-first-time out of 500, versus 50 right-first-time out of 55. So, you do want to know that ratio," she said.

The production metric "can be used to monitor the startup success across products, and then the timeframe needed to reach a mature state," Phillips said. "What's the right-first-time in the first year that you're manufacturing this product, and then what does it look like in year two or year three? And then you can see that maybe it's something that's indicative of your company.

"It might take you two years to say, 'OK, we've got the hang of it. We've got our workforce ready to go,' or you might have a very mature workforce and it's within a product line that you're familiar with. It's an extension or just something a little bit tweaked on a product you already have. So, it can give you an idea of how your company is operating."

The post-production metric has three levels of implementation based on the business needs of the organization and product-risk complexity. The first level involves tracking post-market indicators that should be tracked by organizations anyway, but solidifies these metrics as best practices and provides the metric equations for industry references. The indicators to track are: service records; installation failures; complaints; Medical Device Reports; recalls by number of units involved; and number of recalls.

The second level involves an equation through which to aggregate post-market indicators, resulting in a total post-market score for each product during the time period specified. This can provide a dashboard number that gives a higher-level indication of product quality performance on the market.

Finally, the third level includes a comparative analysis of products through the use of heat maps, dashboards and/or scorecards. This is the highest level of analysis recommended by MDIC to allow senior leaders to keep their fingers on the pulse of the performance of their overall product portfolio.

"The strength of this metric is that it allows for flexibility for companies to decide what the right fit is for them," Phillips said. "It provides a mechanism to foster the discussion against triggering action informing those decisions. You might not see just by viewing complaints on its own as a trend and recalls on its own as a trend, so it does give you a different view.

"This will probably be the most difficult metric for manufacturers to tackle."

[Editor's note:Marla Phillips contributed to this report.]

From the editors of The Gray Sheet