Metrics 'Best Practices' Laid Bare In New Guide That Helps Drive Manufacturers Toward Gold-Star Device Quality

Manufacturers should adopt a "right-the-first-time" approach early on in device development and drive a continual improvement loop across their organizations to reduce risks to product quality, save money and shield brand reputation. That's according to a new manual that helps firms define and apply measurements to strive for ever-enhanced device quality.

Released Aug. 1, the best practices document for quality metrics was developed by the Medical Device Innovation Consortium (MDIC) in conjunction with US FDA and Xavier University. The guidance aims to help firms understand how to use outputs from three specific metrics to inform decisions and prompt particular actions, as well as to aid manufacturers in understanding how to calculate each metric.

"The intent of the FDA/Xavier work was to arm industry with practical metrics to implement commensurate with the needs of the business and complexity of the products, such that the right-first-time mentality could be shifted as close to the initial days of development as possible," the manual states.

Quality metrics are measures used to assess the overall quality of medical device manufacturing. Metrics are an important issue for FDA; the agency's device center included development of metrics in its 2016-2017 strategic priorities list issued in January.

"The goal of a robust metrics program is to help drive continual improvement by enabling an organization to focus on operational areas requiring improvement," the guide notes. MDIC's system of metrics is intended to inform company decisions and trigger action when necessary. Three metrics developed by MDIC address device pre-production, production and post-production actions. The metrics also address industry consideration of how to implement enterprise-wide continual improvement.

The three metrics "need to be assessed along with other metrics and sources of information to provide a more holistic view of the overall risk to product quality," the document states. The purpose of a continual improvement process is to use MDIC's metrics – and other quality measurements – to "enrich knowledge" across an organization.

The guide also illustrates a continual improvement loop that "links the production and post-production phases of the product lifecycle back to the development phase. This feedback loop enables systemic improvements to be made to the rigor of product development." (See figure below.)

Source: Best Practices for Metrics Identified Across the Total Product Lifecycle

Examples of how to implement an organization-wide continual improvement process is included in the document, which points out that it's important for manufacturers to "drive the enterprise-wide review to the lowest points in the organizational structure as possible." It notes that companies should assess data across all products and consistently use that data to make an array of decisions.

For example, "In the case of senior management review, employees should assess and evaluate the data prior to providing it to senior management in order to be able to explain the data context and propose an appropriate course of action for senior management approval," the best practices manual states.

"These recommendations would be further accompanied by an understanding of risk impact to product quality, patient safety, organizational finances and brand equity," it notes. "The data should be provided to senior management with an analysis of the underlying root causes and any contributing causes; a description of what action has already been taken at various stages throughout the TPLC [total product lifecycle]; and a proposal of what other action could be taken to mitigate any additional cost of poor quality."

An enterprise-wide improvement approach calls for a comparative analysis of devices using tools such as heat maps, dashboards and scorecards. "The process represents the highest level of analysis to allow senior leaders to keep their fingers on the pulse of the performance of their total product portfolio," the document states.

MDIC's Quality Metrics Explained Redux

A July Medtech Insight article explained the process for calculating each of MDIC's three metrics. (Also see "At The Intersection Of Quality And Metrics: What's Ahead In FDA’s Effort To More Objectively Measure Quality" - Medtech Insight, 13 Jul, 2016.) The synopsis of how each of the metrics will work is recapped below.

The pre-production metric tracks the number of changes that occurred during the transfer stage that were triggered by product and/or process inadequacies. This metric helps to signal the frequency and volume of changes that could possibly have been avoided by a more robust research and development (R&D) system.

By tracking the metric, the firm has information that can inform the decisions of senior leaders related to potential improvements to the R&D process. For example, upon review, it might be recognized that the rigor of the voice of the customer could be improved, that there could be a more thorough evaluation of literature, or even that improved human-factors studies are needed.

For this metric, it's up to the manufacturer to define "project." For example, does the firm want "project" to indicate the total finished product, or does it want "project" to be the last step in its pre-production process?

"If you then take what you know about each individual project, you can look across your total number of projects," Marla Phillips, director of Xavier Health at Xavier University in Cincinnati and co-chair of MDIC's quality metrics working group, explained at a June MDIC forum. "How many changes are you having, again, based on product and process, across the total number of projects coming up?

"Then you can see, 'OK, from the R&D group that's feeding products to me, how many times are we getting projects from them that require a lot of changing? So it gives you an idea of the R&D group's effectiveness – not just product," she said.

"That's important because if you see a systemic trend, then you can go back and look at how you are gathering development data. Firms can ask, 'How can we make it more rigorous so we don't have these changes?' But again, this has to be commensurate with the needs," Phillips added.

"If you're seeing that you have three changes, or two changes and they're minor, don't drive a huge, company-wide initiative to drive that down and pull resources from things that are actually needed. But you'll know. You'll know when it's too much for your company, and when it's causing problems and costing a lot of money."

And at a separate June meeting of the Association of Food and Drug Officials, Phillips reiterated that one of the pre-production metric's goals "is to help senior management see what's going on, and unless you measure that or have a way to get that bubbled up, it's difficult. You just know people are complaining and that there's a lot of churn, but you just can't put your finger on it. The pre-production metric will help."

The production metric is the "right-the-first-time" measurement that many manufacturers already track. MDIC recommends triaging the root causes such that resources – employees, capital, etc. – can be focused on areas that will result in true improvement, and therefore, a reduction in risk to product quality.

Additionally, MDIC suggests that firms use root cause trend analysis in such a way that any nonconformances related to product and/or process inadequacies are relayed back to R&D through senior management. Again, this type of review will enable the organization to assess the effectiveness of its systems and processes.

"The strengths of this metric are tracking right-the-first-time based on product and process inadequacies," Phillips said. "We can track and trend within and across lots on a rolling basis to identify the highest area of risk. You can apply predetermined action limits. And again, we said we want to inform decisions and trigger action, so is the number good? I don't know. Maybe a low number for one product is actually good. It might even be world-class. So, you have to decide based on your product profile risk what your action limits and trigger limits are.

"The metric is not skewed by volume. However, the volume in this particular case gives you some insightful information, so it is good to know the number of units started because it's very different to say you have 50 right-first-time out of 500, versus 50 right-first-time out of 55. So, you do want to know that ratio," she said.

The production metric "can be used to monitor the startup success across products, and then the timeframe needed to reach a mature state," Phillips said. "What's the right-first-time in the first year that you're manufacturing this product, and then what does it look like in year two or year three? And then you can see that maybe it's something that's indicative of your company.

"It might take you two years to say, 'OK, we've got the hang of it. We've got our workforce ready to go,' or you might have a very mature workforce and it's within a product line that you're familiar with. It's an extension or just something a little bit tweaked on a product you already have. So, it can give you an idea of how your company is operating."

The post-production metric has three levels of implementation based on the business needs of the organization and product-risk complexity. The first level involves tracking post-market indicators that should be tracked by organizations anyway, but solidifies these metrics as best practices and provides the metric equations for industry references. The indicators to track are: service records; installation failures; complaints; Medical Device Reports; recalls by number of units involved; and number of recalls.

The second level involves an equation through which to aggregate post-market indicators, resulting in a total post-market score for each product during the time period specified. This can provide a dashboard number that gives a higher-level indication of product quality performance on the market.

Finally, the third level includes a comparative analysis of products through the use of heat maps, dashboards and/or scorecards. This is the highest level of analysis recommended by MDIC to allow senior leaders to keep their fingers on the pulse of the performance of their overall product portfolio.

"The strength of this metric is that it allows for flexibility for companies to decide what the right fit is for them," Phillips said. "It provides a mechanism to foster the discussion against triggering action informing those decisions. You might not see just by viewing complaints on its own as a trend and recalls on its own as a trend, so it does give you a different view.

"This will probably be the most difficult metric for manufacturers to tackle."

From the editors of The Gray Sheet