Case for Quality - MDIC · 2020-01-01 · FDA engaged with the Medical Device Innovation Consortium (MDIC) in 2014 to develop an ongoing forum to bring stakeholders across the medical

pg. 1

Case for Quality Voluntary Manufacturing and

Product Quality Pilot Program

Results

pg. 2

Executive Summary FDA launched the Case for Quality (CfQ) in 2011 to elevate the focus of all medical device stakeholders from baseline regulatory compliance to sustained, predictive practices that advance medical device quality and safety to achieve better patient outcomes.

FDA engaged with the Medical Device Innovation Consortium (MDIC) in 2014 to develop an ongoing forum to bring stakeholders across the medical device ecosystem together to collaborate on objectives for the Case for Quality. Through MDIC, it was identified that a different approach for evaluating quality execution was needed to increase the focus on quality not just compliance.

In January 2018, FDA launched the Voluntary Manufacturing and Product Quality Pilot Program (The Pilot) that applied a tailored-approach of the CMMI Maturity Model Rev 2.0 along with complimentary changes in FDA regulatory activities. Details and results from that pilot are outlined in this report.

The Pilot focused on manufacturing and the associated support functions and enabled FDA to evaluate the adapted review approaches on manufacturing submissions required for Pre-Market Approval (PMA) devices.

The Pilot has 46 active participating sites across representative manufacturer sizes, US domestic sites, international sites, and various device-risk classifications. More than 80% of the Pilot participants surveyed indicated that the appraisal had a direct value to product quality and over 90% reported a positive experience with the appraisal.

Impact metrics collected from participants have also shown increases to safety through implementing more manufacturing improvements and reducing defects as well as improving access to treatments through increased production capacity. Participants have reported hundreds of thousands of dollars in operational improvements as well as millions of dollars in revenue opportunities. FDA has noted performance improvements in participating sites and maximizing Agency resources due to the modifications piloted.

The Pilot program has demonstrated the impact and value of collaboration and shifting the focus in the medical device industry beyond compliance. As highlighted in this report, the improvements in organizational quality performance increase the availability and safety of the medical device products that are in the hands of patients and users.

To continue the significant improvements demonstrated to date and continue to develop a nimbler and data-driven regulatory framework that improves safety, it is recommended that the Pilot be converted into full operational program within the medical device ecosystem. An operational program should maintain the balance of flexibility and agility provided by the iterative and rapidly learning framework used in the Pilot. The operational program should also consider sustaining the collaborative oversight model used through the Pilot which includes MDIC, FDA, Industry, Patients/Providers, and Payers. This model engages all stakeholders on seeking win-win solutions and sharing information.

pg. 3

Table of Contents Executive Summary ...................................................................................................................................... 2

Background .................................................................................................................................................. 5

Pilot Operations ........................................................................................................................................... 7

Scope ........................................................................................................................................................ 7

Pilot Guiding Principles ............................................................................................................................. 7

Eligibility and Participation ....................................................................................................................... 7

Eligibility ............................................................................................................................................... 7

Participation Expectations .................................................................................................................... 7

Rules of Engagement ................................................................................................................................ 8

Shared Commitments ............................................................................................................................... 8

Appraisal Practice Areas (PA) ............................................................................................................... 8

FDA Pilot Modifications .......................................................................................................................... 11

Inspection Activities: .......................................................................................................................... 11

Submission Review Activities: ............................................................................................................ 12

Pilot Process Activities and Expectations ............................................................................................... 15

Pilot Process Flow and Timeline ......................................................................................................... 15

Appraisal Activities ............................................................................................................................. 16

Pilot Objectives ....................................................................................................................................... 18

Metrics ............................................................................................................................................... 18

Pilot Evaluation and Results ....................................................................................................................... 21

Learnings and Modifications .................................................................................................................. 21

Pilot Metrics ........................................................................................................................................... 24

Participating Sites ............................................................................................................................... 24

Pilot Adoption Metrics ....................................................................................................................... 26

Appraisal Effectiveness ....................................................................................................................... 26

Appraisal Results ................................................................................................................................ 28

Inspection Data .................................................................................................................................. 34

Submission Modification Performance Data ...................................................................................... 35

Impact and Outcomes ........................................................................................................................ 38

Performance Metrics Experience ....................................................................................................... 40

Additional Scenarios ............................................................................................................................... 42

pg. 4

Multi-Site Appraisals .......................................................................................................................... 42

Follow-up and Future Appraisals ........................................................................................................ 42

Data Analysis and Information Collaboration .................................................................................... 43

Conclusion .................................................................................................................................................. 43

Participant Comments ................................................................................................................................ 44

pg. 5

Background FDA launched the Case for Quality (CfQ) in 2011 to elevate the focus of all medical device stakeholders from baseline regulatory compliance to sustained, predictive practices that advance medical device quality and safety to achieve better patient outcomes. The initiative centered around three principle barriers identified in the white paper, Understanding Barriers to Medical Device Quality1. These barriers were the fact that 1) there was no impactful engagement between stakeholders in the medical ecosystem around quality, 2) there was a collective focus on compliance not quality, and 3) finally, there was no comparative data transparency around quality to enable market drivers to reward high-quality and continuous improvement.

FDA engaged with the Medical Device Innovation Consortium (MDIC) in 2014 to develop an ongoing forum to bring stakeholders across the medical device ecosystem together to collaborate on objectives for the Case for Quality. This forum includes participation from FDA, industry, academia, patient groups, payers, and providers and creates a trusted-space for identifying and acting on issues that inhibit quality performance. Through MDIC, it was identified that a different approach for evaluating quality execution was needed to increase the focus on quality not just compliance. A study benchmarking several industries and maturity models, was performed by Deloitte & Touche, LLC and the MDIC Case for Quality Program: Maturity Model Research Report2 released in June 2015. The study identified the Capability Maturity Model Integration (CMMI) methodology as the most suited for the intended application.

FDA engaged with the CMMI Institute and a team of MDIC participants to evaluate the model, ensure applicability to regulatory requirements, identify weaknesses in the CMMI methodology, and develop a pilot program focused on quality maturity. FDA learned that CMMI Model did not establish a quality system, but rather it evaluated the capability and performance of an established quality system. This meant that there was no need to map the model to existing regulatory requirements, as a compliant quality system would be the starting point in any regulated space. This facilitated developing how the methodology would be applied to a medical device manufacturer without increasing burden or impacting any established systems.

When the MDIC team decided on the CMMI model, FDA received immediate feedback from various sources, highlighting negative experiences and concerns regarding the CMMI process. FDA coordinated with CMMI to benchmark with various companies and industries regarding their use of CMMI. Companies representing software, healthcare, and oceanographic products provided their experiences, challenges, and successes using the CMMI model. FDA also received input from the Department of Defense (DoD). DoD has been using model to manage suppliers for several years and shared significant experience with how it has been applied.

These engagements provided insight into the issues and risks existed with the CMMI model and pitfalls to avoid in using the model and appraisal process. Highlighting a series of consistent themes in how the model was implemented both at the organization level and in oversight resulted in issues or failures. The principal themes were:

1 https://www.fda.gov/media/82284/download 2 https://mdic.org/resource/maturity-model-report/

pg. 6

• Establishing a required level for a benefit created a new compliance model and created unsustainable practices just to fulfill the requirements

• Forcing the implementation of the CMMI model from the top down as a checklist for a quality system increases the burden and results in behavior that is not reflective of the way the work is done.

• Applying the rigorous appraisal method used to achieve a CMMI rating shifts the organizations focus away from continuous improvement to generating artifacts for an appraisal.

These learnings were incorporated by FDA, MDIC, The CMMI Institute, and participating stakeholders into The Pilot development.

FDA also identified opportunities to adapt inspection requirements and certain review activities to support faster improvements within the medical device industry. In considering these modifications, FDA found that they would not only support faster quality improvements, the modifications also provided immediate value and created incentives for industry participation and transparency to the agency.

In January 2018, FDA launched the Voluntary Manufacturing and Product Quality Pilot program3 (The Pilot) that applied a tailored-approach of the CMMI Maturity Model Rev 2.0 along with complimentary changes in FDA regulatory activities. Details and results from that pilot are outlined in this whitepaper.

3 https://www.federalregister.gov/documents/2017/12/28/2017-28044/fostering-medical-innovation-case-for-quality-voluntary-medical-device-manufacturing-and-product

pg. 7

Pilot Operations Scope The Pilot focused on manufacturing and associated support functions. The rationale for the manufacturing focus was based on leveraging FDAs experience with Good Manufacturing Practices (GMP), the ease of identifying objective metrics through manufacturing, and the capability to verify a participant’s compliance history. The focus on manufacturing also enabled FDA to evaluate the adapted review approaches on manufacturing submissions required for Pre-Market Approval (PMA) devices.

Design practices, Supplier Management practices, and Servicing were outside of the pilot scope.

No modifications were developed for 510(k) cleared devices, DeNovo devices, or combination products with a drug or biologic primary mode of action.

Pilot Guiding Principles The Pilot uses the model and appraisal process as a continuous improvement tool and not a new compliance requirement.

1. The Pilot should reinforce trust and collaboration on continuous improvement and safety. 2. It is more important to establish a baseline and demonstrate progression than having a perfect

measure or solution. 3. The Pilot does not require the achievement of a specific maturity level to maintain participation

or to qualify for FDA modifications. Participating organizations have already demonstrated a compliant quality system, the appraisal will be used to improve the performance of that system. Transparency and openness with regards to improvement opportunities are more significant to the organization and FDA than a demonstrated maturity level.

4. FDA modifications will be based on commitment to improvement, transparency, and engagement with FDA and the Pilot program.

5. The appraisal will engage primarily with the staff doing the work and understand how well the current processes and system are executing. No additional burden or preparation should be required for the appraisal and the disruption to operations should be minimized.

6. FDA has already inspected participating organizations and reviewed their procedures and documents multiple times before. The focus of the maturity appraisal should be the people and the execution of the work, not the quality or number of documents available or reviewed.

Eligibility and Participation Eligibility

1. Enrollment is site specific 2. The site must be in good compliance standing (No Action Indicated or Voluntary Action

Indicated classification from FDA inspection or MDSAP (Medical Device Single Audit Program) audit within the last 5 years).

Participation Expectations While participating in the Pilot, the company agrees to:

pg. 8

1. Having appraisal(s) scheduled with the CMMI Institute within a target of 90 days from enrollment.

2. Collect and submit a baseline set of organizational or operational metric data to FDA and provide a quarterly update.

3. Participate in established checkpoint activities with CMMI Institute 4. Be available for real-time consultations with FDA and CMMI Institute.

Rules of Engagement The Pilot was focused on assessing and engaging with participants with a focus on continuous improvement. This required more transparency and trust in the engagement between FDA and participants. As part of the Pilot, FDA still maintained its regulatory focus regarding public safety issues. To facilitate the transparency, trust, and early engagement FDA committed to working collaboratively to address any public safety issues that may occur. That commitment was dependent on participants also committing to resolution of any issues. A set of rules were identified for engagement on these issues:

1. Participants and FDA commit to early interaction when safety issues arise or are identified.

2. FDA will meet with participants and commit to identifying and prioritizing the fastest and most effective solutions for the issues.

3. FDA and the participant will establish an action plan and periodic check-ins for visibility into progress

4. If participants do not communicate or act to resolve issues, FDA may remove the participant from the Pilot and engage in additional compliance or enforcement actions.

Shared Commitments • The Pilot is developing and testing a learning system. Feedback, engagement, and transparency

will be encouraged and expected throughout the duration. • Actions, activities, and changes will be guided by what delivers the optimum results for quality

and patient outcomes. The Pilot will consider and test novel approaches if patient safety is assured and legal and regulatory requirements are satisfied.

• The Pilot operation will adapt from feedback and learnings quickly. • The Pilot will incorporate least-burdensome principles into activities. • Metrics and data gathered will be used for learning, not for creating enforcement actions. • FDA and Participants commit to solution-focused interactions with early engagement and

transparency.

Appraisal Practice Areas (PA) Due to the manufacturing focused scope of the Pilot, eleven practice areas of the twenty-five available in the CMMI Development Model Version 2.0 were identified to be covered during the appraisal. The practices areas in scope are indicated in Table 1.

Practice Area Rationale for Inclusion CM (Configuration Management) ENABLING Category

Errors can occur due to incorrect or outdated work material, instructions, or processes. Including this practice

pg. 9

Practice Area Rationale for Inclusion Manage the integrity of work products using configuration identification, version control, change control, and audits.

area provides insight into how well an organization manages version control, document control, and traceability. These changes directly impact the organizations ability to deliver the correct version of the product to its customer.

EST (Estimating) MANAGING Category

Estimation provides a basis for making commitments, planning, and reducing uncertainty. Including this practice area provides insight into how well an organization develops, uses, and improves estimates. This allows for early determination of the duration, cost, and efforts required to implement process improvements or corrective action, and increases the likelihood of meeting objectives including quality and safety.

Estimate the size, effort, duration, and cost of the work and resources needed to develop, acquire, or deliver the solution.

GOV (Governance) IMPROVING Category

Leadership and their commitment to quality is critical to establishing a culture of continuous improvement and patient safety. Including this practice area provides insight into how well an organization’s leadership engages and supports those processes and activities that contribute to the success of meeting the business objectives including quality and safety.

Provides guidance to senior management on their role in the sponsorship and governance of process activities.

II (Implementation Infrastructure) IMPROVING Category

Continuous improvement is key to an organizations ability to consistently achieve goals and objectives. Including this practice area provides insight into how well an organization supports continuous improvement of the processes to ensure they efficiently and effectively achieve objectives including quality and safety.

Ensure that the processes important to an organization are persistently and habitually used and improved.

MPM (Managing Performance and Measurement) IMPROVING Category

Focusing management and improvement efforts on cost, schedule, quality, and safety performance can maximize business return on investment and patient outcomes. Including this practice area provides insight into how well an organization has established metrics, analysis capability, and performance objectives (including quality and safety) that enable leadership to align the organization, measure success, and quickly correct if issues occur.

Manage performance using measurement and analysis to achieve business objectives.

MC (Monitor and Control) MANAGING Category

Actively identifying and monitoring what is important for evaluating progress and success increases the probability of meeting objectives (including quality and safety) by taking early actions to adjust for significant performance deviations. Including this practice area provides insight into how well an organization identifies critical attributes to monitor and the organizations ability to respond and correct.

Provide an understanding of the project progress so appropriate corrective actions can be taken when performance deviates significantly from plans.

PLAN (Planning) MANAGING Category

Effective planning helps organizations and leadership estimate resources, identify critical attributes to monitor,

pg. 10

Practice Area Rationale for Inclusion Develop plans to describe what is needed to accomplish the work within the standards and constraints of the organization

capture what is needed to achieve objectives, and identify organizational gaps that need to be addressed. Including this practice area provides insight into how well an organization optimizes cost, functionality, and quality to increase the likelihood of meeting objectives (including quality and safety).

PQA (Process Quality Assurance) DOING Category

High-performing organizations implement systemic mechanisms for establishing the right process activities, ensuring the processes are adhered to, and continuously improving the processes as needed. Including this practice area provides insight into how well an organization increases the consistent use and improvement of the processes to maximize quality.

Verify and enable improvement of the quality of the performed processes and resulting work products.

PI (Product Integration) DOING Category

Once a product is developed and designed, the product needs to be successfully transferred into manufacturing and integrated into the manufacturing and quality operations. Including this practice area provides insight into how well an organization executes on that design transfer and ensures quality and safety objectives providing a product solution that meets or exceeds the functionality and quality requirements.

Integrate and deliver the solution that addresses functionality and quality requirements.

RDM (Requirements Development and Management) DOING Category

Establishing clear, specific, and meaningful requirements increases the ability to meet customer needs, minimize errors in design, and minimize errors in manufacturing. Including this practice area provides insight into how well an organization solicits and converts all the customer needs, clinical and safety risks, and additional stakeholder needs into clear, actionable, and repeatable work products that ensures that customer's needs and expectations are satisfied.

Elicit requirements, ensure common understanding by stakeholders, and align requirements, plans, and work products.

TS (Technical Solution) DOING Category

Ensuring products can be effectively and efficiently manufactured and fulfill or exceed quality objectives is key to achieving business success. Including this practice area provides insight into how well an organization provides cost-effective design and product solutions that meets customer requirements and reduces rework.

Design and build solutions that meet customer requirements.

Table 1: Selected Pilot Practice Areas and Rationale Practice Area Considerations The Pilot development considered several factors in selecting the Practice Areas in scope. These included:

• Disruptions often introduce unnecessary variability into established processes and result in quality errors. The appraisal should be completed in 5 business days and minimize impact to the business and disruption to the organization.

pg. 11

• The Pilot appraisal can be done in a tailored application of the maturity model, like how FDA performs routine surveillance inspections, without an exhaustive assessment of the entire organization using the complete maturity model.

• The CMMI Maturity model is a highly integrated system and was developed that way from inception. For example, the Managing Performance and Measurement (MPM) Practice Area is related to both Estimating (EST) and Monitor and Control (MC). The relationship of these three areas creates an inherent verification during the appraisal as any inconsistencies between how execution is demonstrated will be clearly observable by the appraisal team. On the other hand, the Requirements Development and Management (RDM) and Configuration Management (CM) Practice Areas both relate to Verification and Validation (VV). Evaluating RDM and CM provides insight into how VV is performed even though the appraisal team is not focused on assessing that area.

In the medical device industry, assessing and managing risk is a foundational requirement for assuring safety. The CMMI Maturity Model includes a Practice Area titled Risk and Opportunity Management (RSK) with the intent of assessing the organizations ability to identify, record, analyze, and manage potential risks or opportunities. This Practice Area was not included in the Pilot for the following reasons:

1. The Pilot was established with organizations who have had a compliant regulatory history. This assured that the organization has already had experience with the risk management required of the regulations.

2. FDA has already evaluated the organizations risk assessments as part of product reviews and new products would still receive the same review.

3. The manufacturing focus of the Pilot evaluated how the output of the risk management process was implemented into the process and product specifications. The Requirements Development and Management practice area was better suited for evaluating that implementation.

FDA Pilot Modifications Inspection Activities: All participants are made aware of and understand that FDA will still perform “For Cause” or Directed Inspections due to safety signals or as needed for follow-up regarding safety events (e.g. Class 1 Recalls).

Activity When Modification Starts Rationale Forgo routine surveillance inspections for participating sites

Modification in the FDA systems is made once the site enrollment is accepted by FDA and the site has scheduled their CMMI appraisal.

• Participating sites have already demonstrated compliance with the quality system regulations verified by FDA through inspection. This is a requirement for the site to enroll in the pilot.

• Pilot is assessing effectiveness of the system and driving improvements Participating site is

removed from risk-based workplan

pg. 12

Forgo PMA preapproval inspections for participating sites

Modification is applied once the appraisal is completed and FDA has received summary and benchmark data.

• Inspections can disrupt process and shift focus from improvement efforts for immediate remediation efforts

• Quarterly pulse-checks and metrics provide more frequent and robust oversight

Table 2: Pilot Modifications - FDA Inspection Activities Submission Review Activities: The target review timelines proposed by the Pilot modifications will be used to drive improvements in the process. FDA commits to process improvements, identifying constraints, and transparency on issues encountered towards those goals. The target timelines may change as the pilot progresses and individual reviews may be impacted by resource constraints and unanticipated issues during the review.

30-Day Manufacturing Change Notices Activity When Modification Starts Rationale

• Streamlined submission process

• Structured data elements in template

• Summary of results for review and reduce redundant submission artifacts

• Capability to bundle multiple changes in one submission

• Submission review and acceptance target:

• 5 Business Days

Modification is applied once the appraisal is completed and FDA has received summary and benchmark data

• Appraisal and metrics provide objective assurance of quality system capability and sustainment with frequent transparency

• Accelerates improvement implementation at participating manufacturing sites

• Reduces artifact generation and provides a focused submission for review

• Allows for trending and learning by FDA

Table 3: Pilot Modifications – 30-Day Manufacturing Change Notices

180-Day Site Change Supplements Activity When Modification Starts Rationale

pg. 13


• Structured data elements in template

• Review of results and reduce redundant submission artifacts

• Submission review and approval target: o 10 Business Days

Modification is applied once the appraisal is completed and FDA has received summary and benchmark data

• The current (approved site) and the new (proposed site) have demonstrated through FDA inspection that the methods used in, and the controls used for, the design, manufacture, packaging, labeling, storage, installation, and servicing are in compliance with the 21 CFR 820 regulations.

• The modified submission focuses on the new facility, and the product and processes impacted by the move.


• Accelerates product manufacturing at sites with verified capability and quality focus

• Reduces artifact generation and provides a focused submission for review

Table 4: Pilot Modifications – 180-Day Site Change Supplements

Original Premarket Approval Application (PMA) and Modular PMA– Manufacturing Submissions Activity When Modification Starts Rationale


• Product specific results for review and reduce redundant submission artifacts

• Focus on critical attributes, results, and the associated monitoring practices

• Forgo preapproval and post market inspection

Modification is applied once the appraisal is completed and FDA has received the aggregate results and performance data

• The facilities, the methods used in, and the controls used for, the design, manufacture, packaging, labeling, storage, installation, and servicing have demonstrated compliance with the 21 CFR 820 regulations through FDA inspection. o Manufacturing facilities with

no FDA inspection history will be inspected prior to FDA approval.

• Reduces artifact generation and provides a focused submission on product and process specific

pg. 14

results, and a deeper dive in supplier controls.


Table 5: Pilot Modifications – PMA Original Submissions – Manufacturing Review

pg. 15

Pilot Process Activities and Expectations Pilot Process Flow and Timeline

Figure 1: Pilot Process Flow Description

Organization applies CMMI processes FDA approves application

5 days from application to enrollment

Within 90 days of enrollment Within 60 days of enrollment

Appraisal executed, and results provided to

organization

PMO, appraisal team, and organization determine appraisal scoping, scheduling, and

plan; performance metrics collection form

Within 30 days after approval Quarterly after approval

PMO reviews appraisal data and anonymizes data for reporting and

trending

FDA is provided the aggregate

results and performance

Annu

al a

ppra

isal

Checkpoint to discuss and validate questions or concerns regarding improvement efforts

Manufacturing submission benefits begin

Routine inspection benefit granted

Organization reports performance metrics , derived measures shared with FDA

pg. 16

Appraisal Activities The CMMI Maturity Model allows flexibility in the practice areas assessed, the rigor of the assessment, and the methodology. The tailored application of the model was designated the Medical Device Discovery Appraisal Program (MDDAP) by the CMMI Institute. MDDAP uses the discovery approach for the appraisal (Evaluation) as opposed to the more rigorous audit (Benchmark) approach applied when an organization pursues a maturity rating.

Appraisal Team Requirements • Lead appraisers in the Pilot must have completed the rigorous CMMI certification process for

lead appraisers. • Lead appraisers for the pilot must have participated on at least 30 appraisals with a

demonstrated history of performance in their quality review record. • The appraisal team must have a minimum of two people to reduce subjectivity. • The team will be trained on the MDDAP process, Pilot expectations, rules of engagement, and

the data collection tools for the Pilot. • Each team will be chosen to ensure significant experience in the medical device industry and

medical device development.

Planning • Appraisal team is selected. • Lead appraisers and the CMMI Institute engage in detailed scoping meetings with the executive

sponsor to learn about the organization, products, and activities performed to determine the sampling strategy for the appraisal.

• Scheduling and logistics planning are performed with the site coordinator. • The appraisal plan is created and submitted for approval. • Performance measures/metrics collection form is shared with the participant. • Pricing is established.

On-Site Appraisal • Discovery appraisal - A comprehensive engagement with all levels of the organization

responsible for the work product being evaluated. The appraisal uses interviews and direct observation to affirm the process and activities are successful, repeatable, and persistent. The appraisals are performed in a non-attributable fashion with an open dialogue centered around improving.

• Objective evidence, including some document review as needed. • Opportunities identified are then shared back with the organizational team to validate the what

was observed. This allows for new evidence to be presented or obtain concurrence from the organization regarding what was identified.

• Presentation of results to the organization. • Development of the heatmap

Post Appraisal Activity • Report finalization and quality check by the CMMI Institute • Submission of results and data to FDA • Improvement planning by the organization

pg. 17

• Scheduling and performing Checkpoint follow-ups

Checkpoints • Checkpoints are performed at regular intervals to review progress of the organization • Update and submit performance measures

pg. 18

Pilot Objectives The Pilot was structured to operate in an agile format with frequent stakeholder engagement and the ability to adjust quickly to identified issues or learnings. The following objectives were used to guide metrics captured, feedback collected, and data analyzed to monitor progress and make operational changes.

• Evaluate interest and adoption of the Pilot • Evaluate if the maturity model and the appraisal approach were complimentary to regulatory

requirements and did not add additional regulatory burden. • Evaluate the applicability and flexibility of the maturity model and appraisal to a variety of

organizations. • Determine if the maturity model can objectively:

o Establish the current state of maturity across the medical device industry o Establish a baseline for organizational maturity performance o Provide granularity to observe differences among participants o Demonstrate progress over time

• Evaluate the capability to collect and use organization or product metrics to demonstrate sustained organizational performance or improvements.

• Develop and evaluate adapted FDA engagement and review approaches • Evaluate the target timeframes established • Capture and evaluate costs, value, and impact of pilot.

Metrics FDA, MDIC and participating companies are interested in understanding the effectiveness of the Pilot and to what extent the Pilot generated value for participants, patients/providers, and FDA. The metrics cover program adoption and effectiveness, value to participants and the FDA, and the trending of maturity levels for each company.

The Pilot collected operational and performance metrics to evaluate how to establish a broader set of standard metrics that allow for benchmarking and analysis across the medical device industry.

The initial Pilot metrics are:

• Appraisal Scores o Summary Scores – Submitted to FDA. Used to establish a benchmark and drive

improvement. o Detailed Practice Level Scores – Granular scoring of the Practice Areas and

organizational performance. Submitted to CMMI for aggregate trending. • Pilot Adoption

o Facilities Enrolled – Number of actively enrolled facilities o Appraisal Executed – Number of appraisals completed o Time to Appraisal – Number of days from enrollment to appraisal start o Appraisers in Program – Number of CMMI Lead Appraisers trained in MDDAP

• Program Effectiveness – Qualitative survey administered by CMMI

pg. 19

o Value to product quality o Appraisal Value o Experience with appraisal o Conflicts with compliance o Net Promoter Score (NPS)

• Value o Financial/Resource metrics – Industry o Operational Improvement/Quality Improvement – Industry/FDA o Patient Safety – Industry/FDA o FDA Resource Use – FDA

Performance Metrics Objective organizational and operational metrics have been a key focus for several initiatives at FDA. MDIC initiated a workstream and project regarding metrics through 2016. Additionally, during the development of the Pilot, a work stream was added to identify relevant metrics to uses during the quarterly check-ins.

As the workstream progressed, it was clear that metrics relevant to the organization were varied, depended on organizational capabilities, and organizational objectives. Defining a common set of metrics would increase burden on the organization and would not drive the behaviors the organization or the Pilot had intended.

For the Pilot, a template form that focused on identifying the sources of process and product risk as well as, the sources of identified issues was developed for use by participants. Participants were asked during the appraisal to use the risks identified through their development activities. The organization was also asked to provide a count of the open risks, closed risks, open issues, and closed issues at each lifecycle phase (Design, Design Transfer, Manufacturing, and Post-Market). Additionally, the time to resolution and mean time to resolution were part of the template.

Table 6 shows an example of the template developed for metric collection.

Sources 1st Selection 2nd Selection 3rd Selection 4th Selection

Scope Definition

Risk Scheduling Risks Quality Audit Risk Findings

Supplier/ Inventory Risks Product Risk

Issue Product Issues Quality Audit Risk Findings

Supplier/ Inventory Issues

Equipment Issues

Phase Design Design Transfer Manufacturing Post-Market Risk Issue Risk Issue Risk Issue Risk Issue

Total Number

Identified

Open 10 5 25 30 200 132 38 Closed 8 3 25 30 200 125 40

Comments

Range of time to

resolution or closure

365 Days

365 Days

240 Days

180 Days

90 Days

90 Days 10-90

Days

pg. 20

Mean time to

resolution or closure

189 Days

61 Days

178 Days 92 Days 35

Days 110 Days 38

Days

Table 6: Pilot metric collection template

The information in the template was intended to:

• Identify what the sources of information participants leveraged to assess if there was a trend that indicated better or more responsive data sources that correlated with the participants capability to identify issues earlier, respond faster, or were associated with higher maturity capability.

• Learn where in the lifecycle issues were being identified and resolved to inform systemic improvements that could move issue identification and resolution to earlier in the lifecycle.

• Observe whether focusing on improving capabilities that reduce time to resolution would provide visibility into organizational performance and continuous improvement.

pg. 21

Pilot Evaluation and Results Learnings and Modifications The Pilot was developed to rapidly incorporate learnings as more appraisals were completed, data was collected, and participant feedback was received.

The CMMI Institute, acting as a neutral Project Management Organization (PMO) for the Pilot, established and coordinated monthly calls with participants, MDIC, and FDA to discuss what was working well, what issues occurred, discuss how to correct and modify for the next appraisal, and track how the Pilot metrics were trending.

This engagement and rapid feedback provided agility to the Pilot that enabled the collaborative team to implement better solutions within the operating bounds than the original plan had considered. Additionally, the PMO was able to incorporate additional activities to address opportunities and pain points observed.

The following changes were made to the Pilot based on feedback and learning captured.

The Pilot originally committed to forgoing surveillance inspections once the company was evaluated and accepted into the Pilot by FDA. The target for executing the appraisal was 90 calendar days. FDA allotted for additional time to execution for reasonable unanticipated delays and with frequent communication. FDA learned that often there were delays in establishing the appropriate contracts, non-disclosure agreements, and legal reviews. This activity could add an additional 30 days to the targeted 90 day timeframe for appraisal execution. If the organization was communicating, this was acceptable to FDA and would be noted for Pilot learning. During the Pilot, two situations occurred where the participants experienced significant delays due to factors outside of the organizations control. FDA modified the Pilot process to start of regulatory modifications once a formal Statement of Work was established between the participating organization and the CMMI Institute to schedule an appraisal.

FDA learned that an additional reason why the maturity model fails resulted when organizations tried to implement the maturity model as a quality system, which resulted in a complex, burdensome, and unsustainable level of effort. The maturity model is a roadmap for improvement and requires an accurate evaluation of the starting point. Therefore, for the Pilot, a decision was made by FDA to not provide details of the maturity model, the appraisal questions, or the appraisal playbook. This prevented participating organizations overinterpreting the intent of the model or adding layers to their systems. The FDA was interested in evaluating the current state of maturity at a participant site with no additional burden or preparation required. With no clarity on what to expect, organizations initially prepared for the appraisals as they would for an inspection or audit. It would then take the appraisal team a day to shift the mindset and establish trust. To address this, the CMMI developed a mentoring program that would partner participants who had gone through the appraisal with new enrollees. This enabled new participants to engage directly with a peer and ask direct questions, learn what the experience was like, and what to expect.

As participating organizations experienced the appraisal and saw value in the approach, interest developed in training their internal staff in the methodology. CMMI established a training module available to participants and FDA to learn about the CMMI Model, the MDDAP method, and receive an initial certification. This training provided additional context for participants on how to apply the model

pg. 22

to accelerate improvement efforts, created more confidence in the approach, and established a pool of resources with medical device experience that could augment appraisal teams.

The Pilot appraisals are at a cost that is incurred by the participating organization. Early feedback indicated that it was a concern and could be prohibitive for small manufacturers. The CMMI Institute responded by implementing a sliding scale cost for manufacturers that were recognized as small businesses by FDA. Additionally, the larger participants offered to have employees that were certified on the MDDAP approach augment the appraisal teams at no cost to the smaller firms. This reduced the out-of-pocket cost for the small manufacturer and provided additional training experience for the participating team member. This activity resulted in enough engagement and value that participating organizations began making it available even beyond the small manufacturers to benchmark best practices, if the participants were not in the same competitive medical device space.

An additional adaption tested and quickly implemented due to unanticipated benefits, was to include a certified employee on the appraisals as an embedded Appraisal Team Member (ATM). The embedded ATM is overseen by the lead appraiser and is required to perform in an official capacity. They are not permitted to respond or guide staff during the appraisal activities. The embedded ATM reduced the cost of an appraisal while providing increased confidence by the organization in the appraisal, providing organization specific context for the results (which improved clarity and action), and increasing accountability for the improvements within the organization.

The Pilot focused on 11 Practice Areas to start and assessed to a level 2 maturity. After the first 6 appraisals, the data showed that two of the Practice Areas selected Technical Solutions (TS) and Product Integration (PI) were consistently satisfied at level 2 by all participants even when there was variability in other Practice Areas. The Pilot development team reviewed the detailed practice levels for the TS and PI practice areas up to level 3 to understand the reason for the pattern. It was quickly evident that the intents associated with the practice levels, were closely aligned with a several of the 21 CFR 820 Quality System elements that FDA reviews, within Design Controls and Production and Process Controls. After discussion among MDIC, CMMI, and FDA, a proposal was made to modify the Pilot to assess TS and PI to a level 3 in future appraisals. The proposal was presented to participants and the MDIC Case for Quality steering committee along with the rationale for the change in appraisal scope. The proposal was accepted and implemented for new enrolled participants.

A significant modification made during the Pilot was regarding the metrics submission. The template and process initially conceived was confusing to participating organizations, added burden, and the lead appraisers were also unable to guide participants as to the intent. After engaging with initial participants, FDA determined that it was more effective for the Pilot to just start receiving the metrics that an organization was actively using to monitor their operations and the performance of the quality system. The change was communicated to participants who had gone through the appraisal and future participants. The adjustment resulted in more engagement and metrics submitted by participants. The metrics submitted were initially focused on what participants assumed FDA wanted and were a series of compliance metrics instead of operational or quality metrics. The CMMI Institute presented to FDA a Performance Benchmarking tool that was used to identify organizational objectives and to trace the metrics and results through to the responsible functions and issues that were inhibiting success. The tool was used to tailor appraisals to find and address systemic causes that impact organizational performance. This was a tool and process that the CMMI lead appraisers were familiar with and could

pg. 23

immediately apply within organizations as part of the appraisal. MDIC, FDA, and CMMI decided to adopt a portion of the Performance tool and align it to another MDIC Case for Quality effort that identified quality domains that were relevant to payers, providers, and patients. The MDIC Product Quality Outcomes Analytics working group defined seven quality domains that were detailed in the Medical Device Quality Outcomes Analytics Report4 released September 2016:

• Safety: Device does not compromise the clinical condition or the safety of patients, or the safety and health of users

• Effectiveness: Device produces the effect intended by the manufacturer relative to the medical condition(s)

• Reliability: Device system or component is able to function under stated conditions for a specified period of time

• Patient Satisfaction: Device is perceived to meet or exceed patient expectations of usability and outcome

• Usability: Device minimizes the risk of user errors by patients or clinicians • Availability: Device is available to fill first request orders • Compatibility: Device is compatible with related devices or drugs, the use environment or

relevant standards

The CMMI Performance tool was modified to focus participants in four quality domains and to capture what metrics they use internally to monitor or drive action in the domains of Safety, Effectiveness, Reliability, and Availability. This change allowed participants flexibility to use what was meaningful to their organization and products while providing context that was meaningful to FDA and other stakeholders. The metrics the Pilot started to collect after this change were more relevant to product quality and organizational performance.

The Pilot was developed as a site-specific activity, with appraisals and benefits applying to the enrolled specific manufacturing facility. Current inspections and auditing approaches utilize this model due to variations in activities and organizational culture that exist even under one global quality system. During the pilot, a test situation was encountered that emphasized current state medical device production often occurs across multiple locations domestically and internationally. This created the need to evaluate the value stream of the production instead of just one manufacturing site. The CMMI model provides the flexibility to perform a value stream assessment and a proposal was developed to test the methodology with one of the participants. The approach was successful and incorporated as an option for participants operating across multiple locations and a multi-site appraisal was conducted under one appraisal activity across the different sites. The length and timing of the appraisal was adjusted to accommodate the additional scope. Companies with multiple manufacturing sites and separate business units could enroll various locations but for the Pilot each site required an individual enrollment application.

4 https://mdic.org/resource/medical-device-quality-outcomes-analytics-report/

pg. 24

Pilot Metrics The metrics reported were current as of June 2019.

Participating Sites The Pilot has been able to enroll and appraise participating sites and test cases with varying characteristics to understand the applicability of the model to a wide array of medical device manufacturers. The enrolled sites vary by size, product type and risk, variability in production, and geographic location.

Table 7 shows the participating site details.

Figure 2: Global Distribution of Pilot Participants

Participant Metrics: Individual Companies: 23 Individual manufacturers participating

(some have enrolled multiple sites) Enrolled Sites: 51 Enrolled and

qualified Active Sites: 46 Sites currently active

US Sites: 30 Active sites Outside US Sites: 16 Active Sites Large/Mid Manufacturer:

40 Active Small Manufacturer:

6 Active

Contract Manufacturer/Sterilizer Sites: 7 Active sites are contract manufacturers or sterilizers

Product Classification Breakdown: (Class of products manufactured at the site)

pg. 25

Class I Only Class II Only

Class III Only

Class I & II Class II & III All Classes

2 Sites 7 Sites 3 Sites 6 Sites 16 Sites 12 Sites Device Classification Panels: (Participants)

Cardiovascular General Hospital Neurology Orthopedic Chemistry

Note: One of the active participants is a site that is not considered a device manufacturer by regulation. The site was enrolled by the parent company due to the improvements observed at other locations using the appraisal method.

Table 7: Pilot Participation Metrics

There was a total of 5 sites that withdrew enrollment from the pilot. The rationale for withdrawing provided are:

• Merger & Acquisition: Firm was acquired, and all planned activities and spending were postponed.

• Business Prioritization: o Firm prioritized participation in the Medical Device Single Audit Program (MDSAP) due

to requirements for their market. o Firm withdrew after Year 1, prior to reassessment due to unanticipated business

demands. The firm reported high value and interest in future engagement once immediate priorities were resolved

• Preparation: Competing priorities kept extending the ability to commit to the appraisal target timelines. The participant was recommended to withdraw until they were ready.

• Value: One participant withdrew after Year 1 due to the limited value the current modifications provided for their organization.

pg. 26

Pilot Adoption Metrics

Adoption of the Pilot within industry can be gauged through the enrollment and sustainment of participants throughout the current operating time. The Participation and Enrollment chart (Figure3) shows increasing engagement and enrollment as the Pilot progressed and demonstrated value. The data accounts for sites that chose to unenroll. Limiting factors for enrollment were driven by

the enrollment criteria established, limited resources within FDA to manage a larger increase during a Pilot, and the limited focus of the Pilot modifications. The Pilot has also shown an 81.5% retention rate based on participants completing or committing to a follow-up appraisal after one year with continued engagement in the Pilot.

Additional adoption of the Pilot can be observed from the independent investment made by participants in supporting the sustainment, such as, number of CMMI certified appraisers trained in the MDDAP methodology, industry staff trained to be embedded ATMs, and FDA staff trained to be ATMs.

CMMI Appraisers in Program Trained ATMs

20 Current 20 Pending

Industry: 37

FDA: 9

Table 8: MDDAP Trained Appraisal Team Members

Appraisal Effectiveness The Pilot collected qualitative data from participants through surveys after the appraisals to measure the effectiveness of the appraisal. The surveys were administered to all individuals who participated in the appraisal by the CMMI Institute. The results are based off 236 survey respondents.

Metric Results Comments

Value to product quality

Yes: 85.4% Percent of respondents that reported the appraisal method provides a direct value to improving product quality

Conflict with compliance No: 98%

Percent of respondents who found no conflict with compliance to regulatory requirements.

Figure 3: Participation Enrollment Trend

05

101520253035404550

Jan-18

Mar-18

May-18

Jul-18

Sep-18

Nov-18

Jan-19

Mar-19

May-19

Jul-19

Participation and Enrollment

Num

ber o

f act

ive

site

s

Months since Pilot start

Retention Rate Year 1: 81.5%

pg. 27

Appraisal has added value

Yes: 94% Percent of respondents that reported the appraisal had provided overall value to the organization.

Experience with appraisal

Positive: 91.5% Percent of respondents that reported a positive or neutral experience with the appraisal. No negative experiences were recorded. Neutral: 8.5%

Net Promoter Score

Recommend: +62 (n=86)

Scored on a scale of -100 to +100 indicating likelihood to recommend to colleague. Scores above +50 indicate high favorability.

Table 9: Post-Appraisal Survey Results Summary

To evaluate the commitment of participants to the Pilot, the time from enrollment to appraisal was measured and tracked. The Pilot had established a target timeline of 90 days, with the possibility of accommodating reasonable extensions with communication.

Average time from enrollment to appraisal

Initial Appraisal: 116 days

One-Year Appraisal: Within 5 days of the one-year mark.

Table 10: Pilot Appraisal Execution Timeline

The target of 90 days was often delayed by the participants established processes for new contracts. Once the CMMI Institute was onboarded into the participant system, scheduling appraisals was significantly timelier, as was shown when the data was separated between the initial appraisals and those occurring at the one-year mark. Sustaining the target of 90 days while accommodating reasonable extensions provided a good balance of visibility into participant commitment and flexibility.

pg. 28

Appraisal Results The results of the initial appraisals are shown in chart 2. The radar plot on the right-hand side shows the aggregated data of the individual sites assessed to date. The results have been anonymized and normalized to the highest practice level achievable in each of the practice areas in scope of the Pilot. The filled areas designated Level 1 and Level 2 show where the practice areas each hit a practice level of 1 and 3, respectively. The maximum achievable score in the MDDAP approach is 90% at the highest maturity level capable in that area. For example, the practice area CM, has a maximum achievable practice level of 2. Therefore, the boundary for level 2 is at 90% in CM. The MPM practice area has a maximum achievable practice level of 5 and the boundary of level 2 for MPM is at 35%. The Pilot only required participants to be assessed up to level 2 for the practice areas in scope. Exceeding level 2 was at the discretion of the participant, except for the TS and PI practice areas. Those two areas were tested at level 3 with willing participants based on high industry performance observed. It is important to note that the performance of a site does not indicate non-compliance or safety issues. The CMMI maturity model exercises the performance of established quality systems and incorporates business process perspectives that are not part of a compliance audit or inspection. All participants underwent a compliance review by FDA before acceptance into the Pilot. FDA does not intend to use the performance in the maturity appraisal to engage in regulatory action. The aggregated results demonstrated the capability of the maturity appraisal to provide granular performance results and more importantly, highlight systemic areas of improvement that could be collaboratively improved, maximizing the impact of agency efforts.

The data observed revealed that the Corrective and Preventive Action (CAPA) system of the regulations implemented by industry was more focused on demonstrating compliance than on problem resolution as intended. This was realized in the collective performance drops in the practice areas and practice levels that drove continuous improvement, such as II and GOV, as well as other detailed practice levels. A collaborative project was initiated by MDIC in 2018 to evaluate the causes and develop solutions to enhance the continuous improvement activities with the medical device industry.

Figure 3: Aggregate Pilot Appraisal Results

64%

60%

54%

90%35%

45%

75%

56%

45%

63%

36%

0%10%20%30%40%50%60%70%80%90%

EST

PLAN

MC

CM

MPM

RDMPQA

GOV

II

PI

TS

Pilot Intial Appraisal Results - Aggregate

Level 2 Area

Level 1 Area

pg. 29

During the Pilot development a concern was raised that the CMMI model and appraisal process was only feasible for large manufacturers and would not apply to small manufacturers with limited resources. The appraisal process and model evalutes the behaviors, outputs, and results demonstrated by an organizations quality system and is agnostic to size. The data collected during the Pilot showed no statistically significant difference in the overall performance

difference between small manufacturers and large manufacturers in all practice areas except for MPM. This may be driven more by the need for large manufacturers to implement more measurement systems to maintain performace objectives than an indication of capability or maturity. Additional data would be needed to determine if the observed difference is systemic across small manufacuterers. While there may be a difference in exectution between small manufacturers and large manufacturers, the maturity model and appraisal process demonstrated both applicability and scalability to both participant categories.

Figure 4: Aggregate Pilot Appraisal Results – Small Manufacturers

0%10%20%30%40%50%60%70%80%90%

EST

PLAN

MC

CM

MPM

RDMPQA

GOV

II

PI

TS

Pilot Intial Appraisal - Small Manufacturers

Level 2 Area

Level 1 Area

pg. 30

Individually, participants used the results to benchmark across sites and specifically target improvement efforts using practices from the higher performing sites.

Figure 5: Manufacturing Site Benchmark Example

Organizations with multiple enrolled sites were able to better identify opportunities at a systemic level and focus resources to close performance gaps. This window provided participating organizations the capability to step-back and strategically improve versus reacting to compliance issues. Additionally, since the appraisal evaluates specifically how the work is being performed and reflects the results back to the organization, it allows for ownership among the staff. This creates increased engagement and accountability in implementing improvements.

The following images show the improvements made by participants through the Pilot year. To accommodate the strategic activities undertaken through the Pilot timeframe, participants who wanted to modify the appraisal practice areas or scope where permitted after submission of a justification rationale developed in conjunction with their lead appraiser and FDA review. Participants were required to maintain GOV, II, and MPM in scope. Participants often chose to pursue higher practice levels or to drop practice areas they had performed successfully in and replace them with other practice areas that better aligned with their operations or where they wanted improvement.

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

EST PLAN MC CM MPM RDM PQA GOV II PI TS

Participant Manufacturing Site Benchmark

pg. 31

Table 11: Follow-up Appriasal Performance Results

Legend

Participant added new practice areas, improved and increased practice level in 2nd appraisal

Practice Areas in Pilot scope Results of initial appraisal

Results of re-appraisal after first year

The box in red demonstrates how to interpret the results in the charts shown.



Shows improvement in practice areas. GOV improved from 31.25% to 61.25%

Shows a new practice area included by participant. When there is no blue bar associated.

Shows practice areas not included at the 1 Year appraisal if there is no green bar associated.

pg. 32





Participant added new practice area, improved and increased practice level in 2nd appraisal

pg. 33




Participant improved and increased practice level in 2nd appraisal


In some cases, as participants improved performance in areas of focus, a decrease in performance was noted in some of the other practice areas. As improvements are made in targeted systems, the changes may stress other systems or processes and expose additional improvement needs. This dynamism is expected and reflects the organizations focus on exposing opportunities for continuous improvements.

By moving the focus beyond compliance and using the CMMI maturity appraisal, participants received more actionable feedback by the appraisal team, which supported strategic plans and investments throughout the organization. The integrated approach of the CMMI model highlighted the impact of all the business processes on the quality performance. Participants

pg. 34

reported that their organizations had been able to shift the mindset from quality is the role of the Quality organization to one where quality was owned by all parts of the business.

The data also provided FDA with a more holistic look at an organization than a traditional inspection. It also provides FDA with new ways of monitoring performance, identifying trends, and engaging in more impactful activities for the ecosystem than pursuing individual compliance actions.

Inspection Data As part of the Pilot, FDA committed to foregoing certain inspections once the participant site was enrolled. Due to delays encountered in the scheduling of the appraisal, FDA modified the Pilot commitment to start the inspection modifications as soon as a signed SOW was established with the CMMI appraisers instead of initial enrollment. This removed the time it took for legal negotiations out of consideration in the enrollment metrics and reduced the likelihood of the process dragging out the appraisal date and the site not receiving an FDA inspection.

The Pilot appraisal process was developed to identify opportunities for improvement and was not applying the rigorous appraisal method used to demonstrate persistent practices and results. Therefore, participation required an annual appraisal. FDA used the results of the appraisal as a risk mitigation to continue foregoing inspections for participants. FDA did not forgo conducting “For Cause” or “Directed Inspections” in response to signals or safety concerns.

A breakdown of the Pilot appraisal metrics is shown below.

Total Appraisals Conducted: 51 Initial Appraisals One Year Appraisals

US Domestic 24 9 International 16 2

Table 12: Summary of Appraisals Performed

The Pilot allowed inspection resources to be more effectively engaged and increased the Agency’s reach, especially in international locations. The Pilot expanded oversight of participant sites in international locations that were not within the Agency resource availability or in locations that the Agency is able to travel to due to safety considerations. Domestically, the Pilot allowed FDA to free inspection resources for higher impact activities. Additionally, the annual frequency required by the Pilot increased the Agency’s engagement and touchpoints with all sites.

The Pilot also waived 4 Pre-Approval inspections that were not assessing a specific technical consideration arising from a PMA product review. This resulted in more predictability for the participant and removed potential impact or delay to the approval timeline. Performance of the manufacturing system for these was assured through review of the product manufacturing results, the Pilot appraisal, the quarterly check-ins, and the quarterly performance metrics.

Two sites did receive a surveillance inspection despite participation at the start of the Pilot. The method of communicating participation in the Pilot to the investigation group had gaps that were not originally

pg. 35

identified, and the inspections were scheduled and announced. Once the inspection resource was committed it could not be adjusted. The inspections were conducted, and no observations were noted.

As part of the Pilot, FDA had communicated that “For Cause” inspections would still be conducted. These inspections serve to investigate a specific signal of a potential safety issue the FDA has detected. The objective is to understand the organizations awareness, investigate the response and corrective actions, and determine if the signal was the caused by a systemic issue resulting from non-compliance or if the signal was unanticipated and the organization responded appropriately. Three “For Cause” inspections occurred throughout the Pilot. The inspections found that the organization’s responses were appropriate, and no non-compliance was noted. FDA follow-up with the investigators within FDA also captured the investigators perspective that notable improvements in operations had been observed at the sites.

Submission Modification Performance Data PMA 30-Day Change Notices For PMA devices, manufacturing changes that may impact safety and effectiveness require notification to the Agency 30 days prior to implementation. During the Pilot development it was observed that this requirement and timeline had an impact on implementation of manufacturing improvements that could reduce defects and improve product quality. The Pilot developed a modified set of review requirements and timelines for PMA 30-Day Manufacturing Change Notices. The submission information was templated to streamline the review and relevant submission data was structured to allow for better analysis. A target goal of reducing the review timeline from 30 calendar days to 5 business days was established to increase adoption of improvements.

Total Received: 98

Percent Reviewed in less than 5 Business Days:

66% 3.1 Day Average

Overall Average: 7 Business Days

Greater than 20 Business Days: 6

Greater than 30 Days: 2

Figure 6: Pilot 30-Day Submission Performance Summary

The structured data from the modified submission increased the FDA’s trending of the review performance and used it to identify internal improvement opportunities resulting from potential bottlenecks, resource issues, or exceptions that impact the review. When there were enough review resources and no process issues, the modified submissions improved review timelines significantly.

Manufacturing changes of streamlined submissions were structured into buckets to characterize the types of changes submitted by Pilot participants. The chart below (Figure 7) shows that 55% of the

pg. 36

changes implemented had either a direct or indirect impact to improving product quality. While cost improvements and supply chain improvements may influence product quality, a direct correlation was not able to be identified to include in this assessment.

22% Cost Improvement

22% Supply Availability

22% In-Direct Product Quality Improvement

33% Direct Product Quality Improvement

Figure 7: Manufacturing Submission Breakdown

PMA Manufacturing Site Change Supplement The Pilot also looked to optimize the 180-Day PMA Manufacturing Site Change Supplement review to facilitate production at manufacturing sites that were demonstrating continuous improvement under the Pilot. These submissions can be resource intensive and infrequent. Applying similar principles used for the 30-Day Manufacturing Change Notices, using data from the appraisal, and the performance metrics allowed for significant structuring and streamlining of the site change submission. The Pilot set an aggressive review target goal to reduce the submission from 180 calendar days to 10 business days. The Pilot has only received three streamlined site changes to date. The review time for those submissions has been 95 days, 23 days, and 13 days, respectfully. The Pilot allows for continuous learning and improvement with each submission, but the original target goal of 10 days may have been too aggressive and unreasonable. It may be necessary to readjust to a more realistic target as learning continues.

PMA Original Manufacturing Review To date, no participants have had a new PMA device to develop or test modified submission with. This is anticipated to be a future test scenario.

It is important to note that development of the submission templates and streamlining was done in collaboration and with feedback from FDA and Pilot participant stakeholders. This optimized meeting the information needs of FDA while providing a least-burdensome method for participants.

0

5

10

15

20

25

Operational Im

prove

ment

Business

Continuity

/Supplie

r

Automation/O

perational Im

prove

ment

Business

Continuity

/Producti

on…

Error-R

eduction/Q

uality

Inspecti

on

Error-R

eduction/P

rocess

Improve

ment

Automation/P

rocess

Improvement

Automation/Yield Im

provement

Business

Continuity

/Producti

on Capacity

Automation/Erro

r Detecti

on

Error-P

roofing/P

roduction Im

prove

ment

Breakdown of Manufacturing Changes Submitted

pg. 37

pg. 38

Impact and Outcomes As part of the Pilot, participants were also engaged to quantify the outcomes of their experience in the Pilot and the modifications in streamlined submissions to evaluate the impact of the Pilot.

Patient/Provider Outcomes

55% of manufacturing changes were to improve product quality and 66% were implemented 21 days faster

This shift incentivized an increase in manufacturer improvement submissions, especially changes to reduce manufacturing defects

882 High-risk patients received treatment and had access to their medical device due the 21-day difference

The improved review timeline resulted in an increase on production capacity which directly drove to more patients receiving necessary treatment than originally capable or anticipated

Increased implementation of manufacturing automation to improve traceability, capacity, and error-proofing

20% of the changes were to implement automation that improved process capability, production defects, process monitoring, and the data quality at participant sites, which enables better products and earlier issue detection

Increase availability of product for treatment of patients in chronic care conditions

Capacity at a participant site increased by 4 times due to improvements and changes resulting from the appraisal.

Defect reductions

Pilot participants reported defect reductions directly related to change submissions which either removed the occurrence of manufacturing defects that needed to be inspected for or significantly reduced the defect occurrence. This eliminates or significantly reduces the risk of quality escapes that could impact patients or providers.

Table 13: Pilot Outcomes Impacting Patients/Providers

Participating Medical Device Manufacturer Impact

Assessment costs • FDA/ MDSAP: $140-

350K – Site operations disrupted

• Pilot appraisal: Less than $80K – No operation disruptions

Pilot participants reported significant savings using the Pilot appraisal with increased value when compared to a traditional compliance inspection or audit. The site also did not experience any disruption in operations throughout the appraisal process.

Reported change notice value examples

• $286 K Annual savings

Pilot participants provided some quantitative value metrics from the modified review activities. Savings gained from operational improvements were often reinvested into the manufacturing activities and support. Several

pg. 39

• 10 Dedicated inspection employees reallocated to higher value operations due to improvement

• 11% Production capacity increase

• Greater than $15 million in product sales

changes increased production capacity and drove increased product sales.

Manufacturing capacity increased by 4 times resulting in increased revenue

Pilot participant reported significant organizational improvements, increased production line investment, and increased hiring to support by 30%.

Strategic/systemic improvement implementation vs compliance resolution

Pilot participants were able to use the appraisal results to proactively target the improvement efforts that would be the most impactful instead of responding to compliance remediation.

Organizational Culture Changes

Participants have also leveraged the appraisal to create a more integrated quality culture and move away from the model of a quality organization that owns performance. They have also reported increased employee engagement and improvement suggestions.

Table 14: Pilot Outcomes Impacting Participants

FDA Outcomes

Improved data-analytics on changes, products, and sites

Improved and granular data allows FDA to look at Pilot participants with a different lens and focus on high-impact activities that improve the industry instead of individual enforcement

Improved submission decision consistency

Modified review enables better traceability of changes all the way from the site, to the product, to the review decision. This allows for improved decision consistency and continuous improvement activities.

Increased staff engagement on process improvement and engagement with participants

Staff has been more engaged in how to improve the quality of the information provided and the review decision. There is increased engagement with participants to

pg. 40

improve clarity, enhance the submission, and improve the review.

Increased resource availability and allocation

The Pilot has allowed FDA to enhance the inspection activities it engages in and where possible, improve the review activities to better focus on the safety and effectiveness.

Greater than $10 Million-dollar savings in annual healthcare costs

Availability and increased capacity of products has resulted in measurable savings to annual estimated healthcare costs.

Best practice sharing among manufacturers

Pilot participants have been sharing best practices and benchmarking among each other. The collective focus is on delivering better product quality and outcomes for the whole system. This has been entirely industry led, with FDA observing.

11 – 46% Quality performance improvement through the participating year

FDA has seen quantifiable performance improvements in participant sites when leveraging the appraisal results and the flexibility to drive the appropriate improvements

Table 15: Pilot Outcomes Impacting FDA Performance Metrics Experience The Pilot had originally designated a method of collecting quarterly performance metrics from participants. The approach was to identify the data sources and data activities that are used to proactively identify issues and resolve them throughout the product lifecycle. The intent was to benchmark and incentivize increased use of data and measurement. This process quickly ran into struggles within participants and lead appraisers were also not accustomed to the process therefore could not guide participants in completing the Pilot template.

A key principle of the Pilot was to start and establish a baseline from which progress can be demonstrated. Since the initial approach was not achieving results, the Pilot adjusted and asked participants to simply provide the metrics their organization used and was already collecting to monitor and respond to issues in their quality system. This resulted in more information from participants and ease in collecting and submitting. There was no prescription of the metric or format.

Table 16 below shows examples of the metrics submitted.

Field Corrective Actions Count of field corrections imitated/open CAPA Effectiveness Percentage of CAPAs CAPA Plan Timeliness Percent of CAPAs completed according to planned

timelines Non-Conforming Events Open and Closed Non-Conforming Events

pg. 41

Continuous Improvement Implementation Number of continuous improvement activities started and closed

Corrective Actions Closed Ratio of Open CAPAs to Closed CAPAs Table 16: Sample of Original Participant Metrics Submitted

These initial metrics were heavily focused on what participants interpreted FDA would want to see and centered around demonstrating compliance. The metrics show sustained compliance but were often reactive and too far downstream to demonstrate proactivity in addressing and preventing product quality issues, organizational improvement, and product quality monitoring.

The Pilot once again shifted and leveraged the Performance Scorecard template used by the CMMI institute and integrated 4 of the 7 quality domains identified by the Product Outcomes and Analytics effort. The initial domains were selected because they were the ones most within a manufacturer’s control. These initial domains were Safety, Effectiveness, Reliability, and Availability (of their medical device). This method leveraged the expertise of the CMMI institute and the lead appraisers and focused the performance metrics in categories that were relevant to FDA and external stakeholders.

The adjusted performance scorecard elements are:

• Quality Domain – The quality domain of the 4 that the measure is associated with. • Quality Objective – A description of the measures purpose. • Business Objective – A description of the business objective that the quality objective is

associated with. This shows the integration of the quality objective with business value. • Measurement – Description of the measurement that is associated with the quality objective. • Level of Measure – The level of the measure (product, product family, across all product

families, across all organization). • Measurement Formula – Description of how the measurement is calculated. • Performance Indicator – Description of how the metric signals good or bad performance. • Limitations – Description of limits or blind spots of the metric • Collected – Description of how the data is collected and frequency • Reported - Description of how the data is reported and frequency • Target – The performance target for the metric • Actual – The actual value for the current reporting cycle.

This change drove a significant difference in the performance metrics submitted by participants. The table below (Table 17) shows the shift in metrics submitted by participants after this change.

Supplier Non-Conformance Rate/Incoming Quality

Maintains quality of incoming material/components

Manufacturing Non-Conformance Rate

Measure process efficiency and identify sources of quality issues for proactive improvement

Product Manufacturing Yield Monitors process quality and efficiency Complaint closure cycle Ensures timely investigation of complaints and driving continual

product quality improvements Employee Safety Measure demonstrates organizational improvements and

proactiveness by improving employee safety

pg. 42

First Pass Yield Identifies process issues early and enables proactive improvements to meet or exceed targets. Improves process capability and product quality.

Damages Financial loss due to issues that reach customers. Reflects the availability of the product and improvements needed in the quality system.

CAPA Conversion Rate Rate of issues that have not been addressed early enough in the manufacturing process to not trigger a CAPA. Monitors the capability of the quality system to detect issues early.

Operational Effectiveness Ratio of true production (actual product manufactured minus defects) over the theoretical maximum production output capability. Demonstrates process effectiveness and proactive improvement activities.

Process Capability Process Capability for critical processes to improve performance and reduce variability.

Scrap Rate Percentage scrap generated to scrap planned/predicted Backorder % of sales yet to be fulfilled Improvement Activity Savings Value and savings of proactive improvement activities

Table 17: Samples of Metric Submission Progress

These recent performance metrics are significantly better indicators of organization sustainment, proactive issue detection, and proactive improvements. These metrics were not going to be compared across participants or for regulatory action. The intended purpose for these metrics was to provide objective evidence of product quality performance. The performance data collected to date has been effective in organizational improvements. The Pilot continues to collect these metrics to inform future appraisal activities for participants and identify objective correlation to product quality.

Additional Scenarios The Pilot has also implemented additional testing scenarios that were outside the scope of the original planning or that were identified for improvement opportunities.

Multi-Site Appraisals During the initial Pilot development, the possibility of evaluating multiple sites at once was discussed. This approach was not incorporated into the initial plans. As the Pilot progressed and stakeholders became familiar with the process and results, a situation arose that required evaluating a cluster of sites where operations occurred across the sites in a specific value stream. A test proposal was developed and discussed with the participants to get acceptance. The test scenario was successful, and the Pilot team has completed 5 multi-site appraisals to date. The approach exercises the work-stream and identifies improvement opportunities in the process. There is ongoing work to develop a method of reporting the results to FDA with enough transparency into the induvial site performances as well as the full value stream.

Follow-up and Future Appraisals Several Pilot participants have selected of their own accord to increase the maturity level for the assessment, include additional areas where they see a need, and to purse the more robust evidentiary assessment method of the CMMI appraisal process. The Pilot has incorporated these changes into the

pg. 43

existing process, especially for participants who are pursuing a third appraisal. There is ongoing effort with FDA to formalize a process that accounts for these modifications to the approach as it increases participant engagement and strategic improvements. This provides flexibility which increases value for the participants and enables improvements to extend beyond the manufacturing focus of the Pilot, resulting in improved quality outcomes.

Additionally, there is discussion on modifying the frequency of the assessment based on data shared and the robustness of the appraisal performed.

Data Analysis and Information Collaboration The Pilot has also identified the opportunity for a collaborative and proactive platform to share organization and product performance data. The focus of this platform will be to drive early issue detection and resolution with a commitment to safety and problem solving. This will enable increased data collection and sharing across the medical device industry.

Conclusion The Pilot program has demonstrated the impact and value of collaboration and shifting the focus in the medical device industry beyond compliance. Participants within the pilot have shown significant capability to improve when there is the flexibility to reflect on the more systemic activities that impact patient safety and business value instead of continuously responding to compliance observations. As highlighted in this report, the improvements in organizational quality performance increase the availability and safety of the medical device products that are in the hands of patients and users. By also acknowledging and connecting the performance to the business value, the improvement efforts at the organization become sustainable.

Despite the improvements demonstrated it should be noted that the medical device ecosystem continues to grow in complexity and issues can and will arise. The collaboration on the Pilot has provided a degree of transparency and commitment from all participating stakeholders necessary for rapidly responding to safety issues and in improving Pilot outcomes. This has increased adoption and trust in the methods and approach of the Pilot among participants. FDA’s commitment to adapting and supporting the improvement efforts has additionally increased engagement and data sharing from the participants, which improves the mission of the agency and the impact of FDA activities. Resources at the FDA to sustain the changes are a significant consideration and will be a factor as the Pilot continues.

The iterative approach built into the effort has enabled rapid improvement and increased engagement. Sustaining this approach will be critical for addressing the opportunities for improvement and expansion with how the Pilot continues to operate.

To continue the significant improvements demonstrated to date and continue to develop a nimbler and data-driven regulatory framework that improves safety, medical device quality, and patient outcomes, it is recommended that the Pilot be converted into full operational program within the medical device ecosystem. An operational program should maintain the balance of flexibility and agility provided by the iterative and rapidly learning framework used in the Pilot. The operational program should also consider sustaining the collaborative oversight model used through the Pilot which includes MDIC, FDA, Industry, Patients/Providers, and Payers. This model engages all stakeholders on seeking win-win solutions and sharing information.

pg. 44

Participant Comments “We have had the opportunity to enrol four manufacturing sites in the FDA Case for Quality Maturity Model program. We’re excited to report that we have found the maturity model appraisal process to be strongly complementary to our existing continuous improvement culture. Completed by an independent third party, the Maturity Model provides a unique opportunity to collaborate with Appraisers to identify areas of focus to aide in the delivery of high-quality products to the patients we serve. We were particularly pleased to find that the Appraisals take place over a five-day period with minimal disruption to normal day-to-day plant operations. This process allowed our manufacturing sites to continue to focus on key priorities while also participating in the appraisal. Our program participation has granted us key regulatory benefits, including a reduction in FDA inspections and reduced 30-Day and 180 Day product transfer notices at each of the participant sites. Already, we have benefited from this reduced 30-Day Notice review cycle with an average of < 5 days for 13 changes; ranging from quality improvements, technology changes, process enhancements and supplier improvements. This expediated review cycle has allowed for greater predictability in implementation plans and faster implementation of changes that will impact product quality. As a part of this process, we have collaborated with FDA to identify 30-Day Notices for inclusion in the pilot. We recognise that many internal processes within FDA are in the process of change, which will be required to implement a scalable long-term solution. Having experienced first-hand the exciting opportunities that this program offers, we are committed to ongoing support of this program and we welcome the opportunity to work collaboratively with MDIC, FDA & CMMI in this regard. “

- Participant: Anonymous “The maturity assessment provided us with an evaluation of the health of our operations, engaging individuals most familiar with our day-to-day work. The assessment helped us identify strengths and weaknesses and opportunities for further consideration. As important, the assessment helped us develop operational excellence metrics that will measure the continuous execution and quality oversight of our processes. Now, a cross-functional team is exploring how we can capitalize on what we learned to further advance our processes and our ability to provide world-class products and services to our customers.”

- Participant: Kathie Bardwell, SVP & Chief Compliance Officer, STERIS Corporation “Innovize has had a great experience going through the MDDAP assessment with experienced and well-trained assessors. The types of concerns found by the assessors provide a great roadmap to help Innovize improve our costs, efficiencies and achieve a higher maturity level.”

- Participant: Mark Rutkiewicz, VP Quality, Innovize

Documents

Case for Quality - MDIC · 2020-01-01 · FDA engaged with the Medical Device Innovation Consortium (MDIC) in 2014 to develop an ongoing forum to bring stakeholders across the medical