Upload
others
View
64
Download
9
Embed Size (px)
Citation preview
ISPE Quality Metrics InitiativeJune 2015
A Report from the Pilot Project – Wave 1
© 2015 International Society for Pharmaceutical Engineering
All rights reserved
ISPE 600 N. Westshore Blvd., Suite 900
Tampa, FL 33609 USA
Tel: 813-960-2105 – Fax: 813-264-2816 – [email protected]
www.ISPE.org
Table of Contents
1 Executive Summary 41.1 Main Findings: Summary 51.2 Next Steps for ISPE’s Quality Metrics Initiative 8
2 Background 93 Pilot Design 13
3.1 Project Governance Model 133.2 Choice of Metrics for the Wave 1 Pilot 153.3 AchievingaStandardizedDefinitionsSet 173.4 Additional Surveys Included in the Wave 1 Pilot 193.5 EstimatesofDataCollectionandSubmissionEffort 203.6 Data Collection Period 21
4 Operational Processes for the Wave 1 Pilot 234.1 McKinsey Operational Process 234.2 Experiences from Participating Companies 24
5 Findings from the ISPE Quality Metrics Initiative Wave 1 Pilot 255.1 Sample Size 255.2 MetricandSurveyDataAnalysisandDiscussion 275.3 Wave 1 Pilot Quality Metric Data Analysis 285.4 Wave 1 Pilot Quality Culture Survey Data Analysis 315.5 DataCollectionandSubmissionEffortData 355.6 Process Capability Survey 425.7 EstablishingStatisticallySignificantRelationships 445.8 StatisticallySignificantRelationshipsinWave1PilotData 465.9 RelationshipsatLowerLevelsofSignificance 535.10 ComparisonsWhereMetricsAreNotDifferentiatedorAreInconclusive 565.11 DiscussionofRelationships 575.12 Complaints Analysis 605.13 Analysis of Product-Based Metrics 615.14 McKinseyAnalyticalEffortandObservations 63
6 Output and Lessons Learned from ISPE Quality Metric Wave 1 Pilot 646.1 Success Factors 646.2 Definitions 656.3 Metrics Collection by Site and by Product 656.4 IndustryEffort 666.5 McKinseyAnalyticalEffort 66
7 Proposals 677.1 RationaleforMetricsProposedasaStartingSetforWave2Pilot 67
8 Conclusions 699 References 70Appendix 1: DefinitionsofQuantitativeMetricsUsedinPilot 72Appendix 2: SurveyQuestions 77Appendix 3: ExamplesforSiteandProductDataCollectionTemplates 79Appendix 4: Case Study Company A 80Appendix 5: DetailedAnalysisofDataandRelationshipsforEachIndividualMetric 84
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 4
1 Executive Summary
ISPE commenced its Quality Metrics Initiative in June 2013 after the US Food and Drug Administration (FDA) announced its Quality Metrics Program in a February 2013 FederalRegisternotice[15]
To assist in the evaluation of product manufacturing quality, FDA is exploring the broader use of manufacturing quality metrics.
Throughanextensiveseriesofengagementswithindustryandotherkeystakeholdersoverthepasttwoyears,FDAhasfurtherindicatedthat“anobjectivesetofqualitymetrics”wouldbereportabletosupporttheirrisk-basedinspectionprogram,asgiveninsections704–706oftheUSFDASafetyandInnovationAct(FDASIA)[9]. The FDA Quality Metrics Program is also intended to move both industry and the agency towardthedesiredstate[18] for pharmaceutical manufacturing. The FDA Quality Metrics Program, including the set of metrics selected, is expected to be published for public comment in 2015.
InawhitepaperdeliveredtoFDAinDecember2013[12], ISPE recommended thatapilotprogramshouldbeconductedwithinindustrytofurtherunderstandtheimplementationopportunities,challengesandbenefitsavailablefromsuchaqualitymetricsprogram.ISPE,incooperationwithMcKinseyandCompany,undertookthisproject.TheresultwastheISPEQualityMetricsPilotProject—Wave1.DesignedanddevelopedbytheISPEQualityMetricsCoreTeam,theprojectdrewontheknowledgeand experience of cross-functional industry representatives, ex-regulators and academicians,withfurtherinsightgainedindetaileddiscussionswithavarietyof industry associations at many industry meetings.
The ISPE Wave 1 Pilot ran from June through November 2014 and included:
f Data collected at 44 sites from 18 participating companies f Datawascollectedretrospectivelyfor12monthsandprospectivelyfor3months
at each site. f AWave1setofquantitativequalitymetrics f Nearlyallmetricscollectedwerereportedatsitelevel;threewerecollectedatproductlevelwithineachsite.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 5
1
1.1 Main Findings: Summary
The Wave 1 Pilot met its overall objectives. A summary of the insights gained include:
f It is feasible to collect and submit a standardized set of metrics. f Themajorityofcompaniesthatparticipatedreportedthefollowingbenefits:
– Gainingadeeperunderstandingofthestandardizedmetricsdefinitionsand design
– Establishing a centralized submissions process trial – Developingaccesstoabenchmarkingreportthatallowedthemtoexamine
their progress against aggregated data from their peers
f Centralcollectionandsubmissionofmetricswillcreateaburdenforindustry,primarilybecausestandardizedmetricswillinevitablydifferfromcurrentcompany metrics.
f Manycompanieswillperformmetricscollectioninadditiontotheirestablished programs.
f Understanding organizational context is crucial to interpreting results. f The Wave 1 Pilot also provided some key insights in relation to the prevailing qualityculturewithinanorganizationthatmeritfurtherexploration.
ThesuccessoftheWave1Pilotcanbetracedtothefollowingfactors:
f Usingastandardizedsetofmetricswithclearandspecificdefinitionsprovidedfor each of the metrics measured.
f Excellentcollaborationbetweenallstakeholders. f Frequentanddirectinteractionforguidanceandqueryresolutionbetween
the McKinsey support team and participating sites. f Leveraging McKinsey’s experience and capability in metrics program delivery. f Leadership from the ISPE Quality Metrics team and the sponsorship from
the leaders at the participating sites. f Ongoingdialogandtrust-buildingwithFDAthroughoutthepilotperiod.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 6
1
TheWave1Pilotalsoidentifiedseveralchallengesthatarepresentinrollingouta centrally reported standardized metrics program. These include:
f Theindustryanditssectorshavenottraditionallysharedacommondefinitions.Consequently,definitionsofeachmetricmustbespecific,clearlyunderstoodand meaningful across the range of organizations under consideration to ensure theprogram’ssuccess.Thiswillrequiredetailedup-frontdesignandongoingoperational support.
f Thelevelofeffortrequiredfordatacollectionandsubmissiononbehalfoftheindustry, and data analysis and support on behalf of the agency cannot be underestimated.RefertoSection 5 and Section 5.14 of the report for estimates oftheeffortinvolved.However,theseestimatesmaybeconsideredconservativebecause they do not include several factors, such as: – A“goodenough”situationwasappliedtodatasubmissionintheWave1Pilot.SubmissiontoFDAwouldrequiremorethoroughandcompletedatacollection,additionalmanagementreviewanddataverification.
– Pilotparticipantshadtheflexibilitytoprovidetheirmostpragmaticdataset(e.g.,allproductsatthesiteoronlythosefortheUSmarket);thiswouldnotbe the case in a formal submission process.
– Pilotparticipantstypicallyhadmaturesystemsandcapabilitiesandwerefromdevelopedcountries;thiswouldnotbethecaseforallsitesundera centralized reporting initiative.
f Understandingthevariationinrangesforinterpretationofthedatawillrequirelonger timeframes to assess than those examined in the Wave 1 Pilot.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 7
1
Details of the statistical analysis, main outcomes and recommendations arising from the Wave 1 Pilot can be found in Section 5 and Section 6 of this report. A summary ofthesefindingsisasfollows:
f Evenwith44sitesreportinginthisinitialphaseoftheISPEQualityMetricsInitiative,thesamplesizesallowedstatisticalanalysiswithsomelimitations;these are outlined in more detail in Section 5.3 of this report.
f TheanalysisoftheWave1Pilotdataidentifiedanumberofstatisticallysignificant(lessthan5%likelihoodofacoincidence)relationshipsbetweenthemetricscollectedandoverallqualityoutcomesatthesites.
f Theseinitialfindingsofstatisticallysignificantrelationshipsdonotimplycausation,thereforethisreportdoesnotattempttodrawconclusionsfromthisphaseofthedataanalysis.Instead,thisreportisintendedtosharethefindings,identifyrecommendationsandputforwardproposalsforthenextphaseofthestudy.
f To meet one of its objectives, this pilot targeted a set of metrics collected at both the site and the product level to understand the current capacity to collect metrics.Product-levelmetricsrequireanunderstandingofthechallengesofaggregating metrics across the supply chain for multisite products. In addition, thedefinitionof“product”asrelatedtoan“applicationnumber”presentedissues for over-the-counter (OTC) products. This element of the pilot has led to some key learnings and recommendations for both industry and FDA, and these are included in the report.
f Quality metrics reporting alone should not be the basis for action (either positiveornegative)withoutunderstandingthecontextofthedataandthe originating company.
f Choosinganappropriatemetricsetwillhelpidentifycontinualimprovement opportunities.
f TheknowledgegainedfromtheWave1PilothasbeenleveragedtodeveloparevisedsetofstartingmetricsthatISPEnowproposesforfurtheranalysisina Wave 2 pilot program. Details of this proposal can be found in Section7 of this report.
f LearningsfromtheWave1PilothavealsobeensharedwithFDAforconsiderationinthedesignofagency’sfinalsetofobjectivemetrics.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 8
1
1.2 Next Steps for ISPE’s Quality Metrics Initiative
BasedonthefindingsfromtheWave1Pilot,ISPEnowrecommendsthefollowingset of starting metrics:
1. Lot acceptance rate (normalized by lots dispositioned), collected at site level2. Lot acceptance rate (normalized by lots dispositioned), collected at product level
withinasite3. Critical complaints (normalized by packs released), collected at product level
byeachproductapplication,notbrokendownbysite4. Critical complaints (normalized by packs released), collected at site level,
undifferentiatedbyproduct5. Deviations rate at site level
FollowingthepresentationoftheWave1PilotresultsattheISPEQualityMetricsSummitinBaltimoreon21–22April2015,itwasbroadlyagreedthatthereisacontinuingappetitewithinindustryforadditionallearningwithrespecttoqualityperformance measures.
ISPE has therefore initiated planning for a Wave 2 Pilot, to commence in the second halfof2015.Thissecondphasewilltestthestartingsetofmetricsonanextendedsampleandtimeperiodtoincreasetherangeanddurationoftheknowledgebaseand enable more in-depth statistical data analysis to examine correlations and dependences.TheWave2Pilotwillalsoexploretheinclusionofotherpotentialmetricsofinterestandfurtherstudyoftheassessmentofqualitycultureatparticipating companies.
Itishopedthatcontinuingthisworkwillenablethepharmaceuticalindustrytoundertakethe“qualityrevolution”[16] proposed by Dr. Janet Woodcock at the ISPE Quality Metrics Summit to truly enhance the future state of pharmaceutical manufacturing.
Sinceregratitudeisextendedtotheparticipatingcompaniesandtheirstaffforthe excellent input, support and enthusiasm given throughout this Wave 1 Pilot.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 9
2 Background
This section describes the:
f ISPE Quality Metrics Initiative background f Quality Metrics Pilot Program, a major component of this project f Wave 1 Pilot background, rationale, and key milestone dates
TheFDA’svisionfortwenty-firstcenturymanufacturinginthepharmaceuticalindustryisoftenquotedasthe“desiredstate”andis:
A maximally efficient, agile, flexible pharmaceutical manufacturing sector that reliably produces high quality drugs without extensive regulatory oversight.
There have been many guidelines issued by FDA [1][2][3][4] and the International ConferenceonHarmonisationofTechnicalRequirementsforRegistrationofPharmaceuticalsforHumanUse—Q8[5], Q9 [6], Q10 [7] and Q11 [8]. These provide a regulatoryframeworkthatallowindustryandregulatorstomovetowardthisdesiredstate.Despitetheintroductionofthenewguidance,recentacknowledgementsconfirmthatmoreworkisrequiredbyboththeindustryandtheregulatorycommunitytoattainthe desired state.
TohelpFDAandindustryworktowardthetwingoalsofensuringproductqualityina global supply chain and reducing drug shortages, FDASIA [9] (July 2012) gave the agencynewauthoritytoenhancethesafetyofthedrugsupplychainandcreatedlegislativemandatesaffectingcurrentgoodmanufacturingpractices(CGMPs).FDASIAalsorequiredFDAtoimplementarisk-basedinspectionprogramofpharmaceuticalmanufacturingsitesratherthanthecurrenttwo-yearinspectioncycleineffect.
Ofrelevancetoarisk-basedinspectionprogramaresections704,705and706ofFDASIArelatingtoadvancedprovisionofinformation(e.g.,qualitymetrics).
f Section704“…enablesFDApersonneltosearchthedatabasebyanyfieldofinformationsubmittedinaregistration…”
f Section705requires“risk-basedschedulefordrugs”andlists“riskfactors”as:(A) The compliance history of the establishment.(B) The record, history, and nature of recalls linked to the establishment.(C) The inherent risk of the drug manufactured, prepared, propagated,
compounded, or processed at the establishment.(D) Theinspectionfrequencyandhistoryoftheestablishment,includingwhether
theestablishmenthasbeeninspectedpursuanttosection704withinthelast 4 years.
(E) Whether the establishment has been inspected by a foreign government or an agency of a foreign government recognized under section 809.
(F) Any other criteria deemed necessary and appropriate by the Secretary for purposes of allocating inspection resources.
Section706requires“…recordsorotherinformation…beprovided...inadvanceorinlieuofaninspection...”These“recordsorotherinformation”areinterpretedasprovisionofqualitymetricsdataaspartofotherinformationwhichcouldpotentiallyberequested.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 10
2
InresponsetotheFDAinitiativeonqualitymetrics,ISPEestablishedaProductQualityLifecycle Implementation®(PQLI)–sponsoredQualityMetricsprojectwithateamthat consisted of representatives from a variety of pharmaceutical companies.
Guiding principles established at the commencement of this project stipulated that anyproposedmetricswouldbe:
f Clearlydefinedtoallowconsistentreportingacrosssites f Objective and meaningful f Easy to capture f Easy to report f Normalizedbyfactorssuchasprocessdifferencesandtechnicalcomplexity f Abletodriveacceptable,notunwantedbehaviors
Thisteamstartedworkatawell-attendedsessionoftheISPE–FDACGMPconferenceinBaltimoreon12June2013,atwhichthebothFDAandindustrywererepresented.TheinitialobjectiveforISPE’sprojectteamwastoanalyzeandusetheoutputfromthediscussionatthismeetingasinputtoawhitepapertobeissuedfordiscussionwiththeFDA.AsummaryofmainmilestonesfortheQualityMetricsPilotProgramisshowninFigure1.
Figure 1: Major Milestones for the ISPE Quality Metrics Project
Quality Metrics Progress
Jan2013 Apr AprJul Oct Jul Oct
Jan2014 Apr Jul
Jan2015
Initialindustryresponses
FDA FR noticerequestingmetrics
Brookingsstakeholdermeeting
FDA metrics of “potential interest”
Kick offISPE QMteam
Pilotkickoff
LauchWave 2?
Pilotdata lock
ISPE QMSummit
Industry whitepapers
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 11
2
TheISPEwhitepaper,publishedinDecember2013,proposedalistofmetricsacceptable to industry that could be reportable to FDA to support a risk-based inspectionprogram.Thewhitepaper’smainrecommendationswereto:
f Conductapilottofleshoutstandarddefinitionsandapproach. f Initiatewithsitemetricscollection,withthepotentialtomovetoproduct
metrics later.
Based on these recommendations, ISPE announced its intention to conduct a Quality Metrics Pilot Program on 12 March 2014.
During this period, FDA expressed a desire for industry input on the development of FDA’squalitymetricsprogram.FDAthenparticipatedindiscussionswithindustryattheMeasuringPharmaceuticalQualitythroughManufacturingMetricsandRisked-Based Assessment meeting held 1–2 May 2014 and hosted by the Engelberg Center forHealthCareReformattheBrookingsInstitution.[10]Anextstepidentifiedatthismeetingwas:
That the pilot quality metrics programs currently under development by the International Society for Pharmaceutical Engineering … may yield important lessons
for FDA as it moves forward with its own program.
In preparation, ISPE explored the options of conducting the Quality Metrics Pilot Programincooperationwithasuitableindependentpartnerthatcouldprovideoperationalexpertiseandassureparticipantconfidentialityduringthepilot.ISPEsubsequentlyagreedtopartnerwithMcKinseyandCompany,duetotheirexperienceofconductingindustry-benchmarkingprograms,specificallythePharmaOperationsBenchmarking of Solids (POBOS) series of programs, [11]whichhavebeenoperatingsince2004.Itwasrecognizedthatbenchmarkingprogramsrequiresignificantexpertiseto succeed, such as:
f Developmentoftemplatestoallowforeaseofinputofdata. f Structureddatasubmissionwithdetailedguidanceonhowtoreportthedata. f Ability to comment on data points to enable interpretation. f Experienceddedicatedsupportforquestionsandclarificationsduringand
throughout the data collection. f Built-indatavaliditychecksandjointreviewtoensuredataconsistency
and accuracy. f Development and operation of supporting IT systems. f Relevanthigh-levelstatisticalexpertisetoassistindatainterpretation. f Autonomyandconfidentialityindatacollection,reviewandanalysis.
ISPEannouncedthelaunchoftheQualityMetricsPilotPrograminpartnershipwithMcKinseyandCompanyattheISPE–FDACGMPconferenceon2June2014.Itwasintendedthatthepilotprojectwouldhavephases:Wave1andWave2.Thisreportsummarizes the Wave 1 Pilot and proposes recommendations for programs and metrics for consideration in a future Wave 2 Pilot.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 12
2
Wave1Pilotwasintendedtodemonstratethefeasibilityandvalueofstandardqualitymetrics.Someimportantprimaryobjectiveswereto:
f Testtheharmonizationofdefinitionsforasetofindustrymetricsthatrepresentboth leading and lagging indicators.
f Testthefeasibilityofcentralizeddatacollectionacrosscompaniesatdifferentmaturitylevelswithintheirowninternalmetricsprograms.
f Exploreindustrypracticesintheareasofqualitycultureandprocesscapability. f Inform continued industry input to FDA.
Industryparticipantswereintendedtogainthebenefitsof:
f InfluencingtheoutputfromtheISPEQualityMetricsPilotProgramintermsofchoiceofmetrics,definitionsandeaseofdatacollectionbasedonactualexperience.
f Receivingablindedcomparisonorbenchmarktotheparticipatingsiteindustryaverageandtosimilartechnologyplatformpeers(providedsufficientsamplesize is achieved).
f Havinganopportunitytodeveloporenhanceinternalproceduresformetriccollectionalongwithasetofmetricdefinitions.
f Gaining insight into the implications of external metric reporting.
Duringtheperiodfromthewhitepaper’sissuetoinitiationoftheWave1Pilot,considerableattentionwasgiventochoiceofmetricstobeincluded.ConsiderationwasgiventodiscussionfromtheBrookingsmeeting,aswellasinputfromFDAandfromotherindustryassociations.ThefinalWave1Pilotmetricschosenwereselectedtomeasureobjectivequalityperformanceofasite.TheyincludeallthemetricsidentifiedbyFDAattheBrookingsmeeting,twotechnology-specificmetricsandtwosurveys,oneonqualitycultureandoneonuseofprocesscapability.MorediscussiononchoiceofmetricsandtheirassociateddefinitionsisgiveninSection 3.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 13
3 Pilot Design
3.1 Project Governance Model
To manage the Wave 1 Pilot project, ISPE and McKinsey established a project governancemodel,whichisshowndiagrammaticallyinFigure2.
Figure 2: ISPE and McKinsey Project Governance Model
Data flow
Communicationand guidance
Communication
FDA Quality MetricsProgram Leader
Quality Metrics Pilot Program Core TeamLeader
McKinsey POBOS Expert
McKinsey partner
Representatives from - Originator Companies, small and large molecules - Generic companies - OTC company
Academic
ISPE staff and advisor
McKinsey Operational TeamData input and analysis specialists
Participant CompanyCompany leadSite leadsSenior leaders
ISPE Quality Metrics Subteams DefinitionsInfluencing/CommunicationQuality Culture
ISPE and McKinsey Sponsor TeamISPE President and CEOMcKinsey PartnerISPE International Leadership Forum of senior leaders representing - Originator Companies, small and large molecules - Generic companies - OTCISPE staff and advisor
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 14
3
Key features of this project governance model are:
f Data from individual companies are seen only by McKinsey personnel f ISPE project team has access only to aggregated data across all companies or tosubsetsofcompanieswherenumbersaresufficienttomaintainanonymity
f The ISPE Quality Metrics Core Team, representing a broad spectrum of the pharmaceuticalbusiness,meetregularly—usuallyweekly.
f The ISPE Sponsor Team consists of ISPE’s president and CEO, senior leaders of pharmaceutical companies in ISPE’s International Leadership Forum and a McKinsey partner.
f The ISPE Core Team seeks communication, guidance and decisions from the ISPE Sponsor Team at about approximately monthly intervals.
f SubgroupsoftheISPECoreTeamheldregularteleconferenceswithparticipantcompany leads and site leads to: – Brief them on progress – Provideanoverviewofthedataanalysisfortheirreview – Seek their input
f SubgroupsoftheISPECoreTeamheldinformalmeetingswiththeFDAQualityMetrics Program leader to: – Seek input to choice of metrics and overall design of the Wave 1 Pilot – Provide update on progress of the Wave 1 Pilot – Provide early readouts of summary Wave 1 Pilot results – Bepresenttosharefindings
Inaddition,theCoreTeamtaskedsubteamswithchartersanddeliverablestoprogressparticularelementsoftheprojectindependently.Subteamswereestablishedfor:
f Definitions f Communications f Influencingandindustryengagement f Quality culture f Process capability
Theimportanceofdefiningmetricscarefullywasidentifiedearlyintheproject.TheDefinitionsSubteamdevelopedtheworkdescribedinSection 3.3.Itwasespecially key to:
f Developingresponsestofrequentlyaskedquestions(FAQs) f ProducingthedefinitionsgiveninAppendix 1 f Leading the process to develop the surveys given in Appendix 2.
TheInfluencingandIndustryEngagementSubteam’srolewastoencouragecompaniestoenrollfortheWave1Pilotandtoarrangeteleconferenceswithpilotleadindividualsand senior leaders in participant companies. Additionally, this subteam took the lead in preparing material for presentation at the FDA meetings.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 15
3
TheQualityCultureSubteamexplorednewideasandpotentialleadingqualitymetrics.GiventhefindingsfromtheCoreTeamandthelevelofpublicinterestexpressedatmanymeetings,thissubteamfocusedonsharingcurrentqualityculturebestpractices.TheteaminitiallyplannedtoexplorewhetheraquantitativeQualityCultureIndexcouldbeestablished.Subsequentwork,however,includingdiscussionandengagementacrossindustryandacademia,hassuggestedthatqualitycultureevaluationrequiresaholisticapproach,andcentralizedreportingofastandardizedassessment is not desirable. The Quality Culture Subteam is developing a cultural excellenceframeworkentitledThe Six Dimensions of Quality Culture;futurepublicationsare also planned.
TheProcessCapabilitySubteam,workingunderthewiderPQLIumbrella,wasestablished based on a recommendation from the Quality Metrics Core Team. Its objectivesaretoproduceaseriesofarticlesand/orwhitepapers,aswellascasestudies, a potential baseline guide and industry sessions at ISPE meetings to examine the use of process capability measurements by the pharmaceutical industry globally. Thisteamalsocontributedtotheprocesscapabilitysurveyquestionsconductedas part of the Pilot Wave 1.
3.2 Choice of Metrics for the Wave 1 Pilot
This section outlines the list of metrics used in the ISPE Quality Metrics Pilot Program, Wave 1andtheirassociateddefinitions.
The choice of metrics for the Wave 1 Pilot evolved over time by taking account of manyinfluencesandinput.OutputfromtheBrookingsmeetingwasconsideredbytheISPEprojectteam,aswellasFDA’srequestforproduct-basedmetrics.Therewasalsoastrongdesirebymostpartiestostarttounderstandtheimpactofqualityculture.Additionallytwotechnology-specificmetricsforsterileproductmanufacturewereincluded.AsummaryofthelistofmetricsusedintheISPEQualityMetricsPilotProgram, Wave 1andtheiroriginsisshowninFigure3.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 16
3
Figure 3: Summary of Final Metrics Collected During ISPE Industry Wave 1 Pilot
Quantitative metrics Technology specific metrics
Additional survey-based metrics
Media fill (for sterile aseptic sites) failuresEnvironmental monitoring (for sterile aseptic sites)
Process capabilityQuality culture
Consensus Industry Metrics
Metric proposed in Brookings meeting
Product and site-based metric
Lot acceptance rateComplaints rate (total and critical)Confirmed OOS rateUS recall events (total and by class)Stability Failure rateInvalidated (unconfirmed) OOS rateRight first time (Rework/Reprocessing) rateAPQR reviews completed on timeRecurring deviations rateCAPA effectiveness rate
2 more quantitative metrics calculated from data collected for Wave 1:• Deviations rate• Incoming material OOS
3.2.1 A Note on Product-Based Metrics
Thefollowingmetricswerecollectedonbothsiteandproductbasestohelpgainanunderstandingofthedifferencesbetweentheseapproaches:
f Lot acceptance rate f Total and critical complaints rate f Confirmedout-of-specification(OOS)rate
Product-basedmetricsintheWave1Pilotwerecollectedonrelevantunitoperationsperformedataspecificsite.Forthispilot,metricswerenotaggregatedacrossmultiplesitestothefinalpackagedandlabeleddosageformlevelortoanewdrugapplication(NDA) level.
Whenreviewingproduct-levelmetricsreporting,it’simportanttoensurethatdefinitionsandexpectationsareclearlydefinedandunderstood.USNDAs,forexample,caninclude multiple dosage form strengths, and each strength may be assembled into aseriesofpacks.ThismeansthatoneNDAmayincludemany“products”–ifa“product”isdefinedasonedosageformstrengthassembledintoonepack.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 17
3
3.3 Achieving a Standardized Definitions Set
Experience from the project team, feedback at ISPE public meetings and preparation ofthewhitepaperallidentifiedtheimportanceofdefiningmetricsthat:
f Are clear to the project team. f Are understood by participants. f Match those currently used by companies as closely as possible. f Measurequalityperformanceaccurately. f Reducetheopportunityfor“gaming”. f Minimizeunintendedconsequences.
TheDefinitionsSubteamestablishedathoroughandrobustapproachtoderivethedefinitionsusedintheWave1Pilot,usinganiterativeprocesstoreachconsensus.This process is depicted in Figure 4.
Figure 4: Process to Derive Definitions for Wave 1 Pilot
ISPE Dec 2013 white paper
Brookings meeting
POBOS quality definitions
ISPE Quality Metrics Definitions subteam
Metrics definitions examples from individual companies
ISPE Quality Metrics Core team
Templates with definitions of each
data point
Inputs from multiple broadindustry sources
Development of precise definitions for each data point
For review Feedback
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 18
3
TheDefinitionsSubteamreceivedinputsfrommultiplesources.Outputswerecollatedintoasetofagreed-upondefinitionsintheExceldata-collectiontemplatesproducedby the McKinsey team, then given to participating companies for completion.
Oneexampleofthiscomplexseriesofinteractionscanbeseeninthedefinitionof“lotdispositioned.”Thisisacriticalterm,whichrequiredconsensus,asitisthe denominator for several metrics collected in the pilot. A representation of this consensus building process is given in Figure 5.
Figure 5: An Example of Challenges Defining “Lot Dispositioned”
"Total number of lots for commercial use produced
and/or packaged on site that went through final disposition
during the period, i.e. were released for shipping or
rejected (for destruction). Rejections should be counted as final disposition regardless at what production stage the rejection occurred. Release is only final release for shipping. Excludes lots that have been
sent for rework or put on hold/quarantined in this
period and hence are not finally dispositioned. Excludes
lots that are not produced or packaged on site, but
released for CMOs."
Final definition of "lots dispositioned"If several formulation lots are aggregated in one “combo” packaging lot, and it is rejected at the packaging stage, does it count as 1 or several rejections?We count every batch rejected as a rejection, since it counts rejections at any stage of the production process but in the denominator as the final lots dispositioned.
We do release work for other sites, but do not produce those batches. do these count?Excludes lots released for other entities, not produced on site.
Do we count intermediate batches?No, only batches shipped unless we ship intermediates.
If a formulation is split into several packaging lots, does each lot count as a new "attempt" or does it remain counted as 1 attempted lot?Yes it counts as multiple lots, Splitting or aggregating of lots happens – it only matters what is finally dispositioned not earlier changes.
Do we count products that were produced in previous years and we now release?Yes, counts on holds or rework in the period when finally dispositioned.
?Even with standardized definition, there will be remaining unclarities (e.g., what do we do with continuous manufacturing?)
Inadditiontodevelopinganddesigningdefinitions,theDefinitionsSubteamalsoassessedanyquestionsraisedbyparticipantsduringthepilotkickofforintheearlyphasesofcompletingthedata-collectiontemplates.ThesubteamissuedaweeklylistofclarificationsintheformatofaFrequentlyAskedQuestionsdocument(FAQs)to all participants.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 19
3
WhileanalyzingdataintheWave1Pilot,itbecameapparentthattwoadditionalmetrics – deviations rate and incoming material OOS rate – could also be calculated automatically,sincethedatarequiredforthesemetricswasalreadybeingcollected.This brought the number of metrics analyzed to 14.
DefinitionsforallquantitativemetricsusedintheWave1PilotaregiveninAppendix 1 of this report.
3.4 Additional Surveys Included in the Wave 1 Pilot
TwoqualitativesurveyswerealsoconductedaspartoftheWave1Pilot.Theseexploredtheprevailingqualitycultureatthesiteandexaminedtheuseofprocesscapability monitoring and trending on the site.
3.4.1 Quality Culture Survey Development
UsingMcKinsey’spreviousexperienceinconductingqualityculture–typesurveys,theDefinitionsandtheQualityCultureSubteamsreviewedanexistingPOBOSQuality Culture Shop Floor Survey for inclusion in the Wave 1 Pilot. Using this survey, thesubteamsdevelopeda15-questionassessmenttoolthatmeasuredfiveculturalelements: Leadership, Governance, Integrity, Mindset and Capabilities.
Althoughcompletingthisqualityculturesurveycouldbeasubstantialamountofworkforparticipants,itwasconsideredanecessarytooltotestthehypothesisofhowqualityculturemayimpactthequalityperformanceoutcomesatagivensite.
3.4.2 Process Capability Survey Development
PreviousworkundertakenbytheISPEQualityMetricsCoreTeamandadditionalreviewwithindustrycolleaguesindicatedtwothings:
f Applicationofprocesscapabilitymeasureswasnotwidespreadinindustry f Sitesthatdidmeasureprocesscapabilityusedawidevarietyofapproaches
Itwasdecided,therefore,todevelopasurveytoassessthetoolsandprocessesused to monitor process capability by the participating sites.
Both the Quality Culture Survey and the Process Capability Survey are included in Appendix 2 of this report.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 20
3
3.5 Estimates of Data Collection and Submission Effort
Companieswereaskedtoestimatetheperson-hoursofeffortthattheyusedtosetupandcarryouttheirpartoftheWave1Pilot.Eachsitecompletedatemplatewithestimatesofthetimeandeffortspentcollectingeachindividualmetric:
f Time (hours) spent to collect individual metric data at both site and product levels, for both the retrospective and the prospective periods.
f Degreeofdifficultyonascaleof1to4(easiesttomostdifficult)forcollectingeach metric at both site and at product levels.
f Siteratingsofwhetherthedatawasavailableintherequestedformorrequiredrecalculation/aggregation,orwascollectedfromfragmentedsources.
Companiesdidnotindicatehowmucheffortwasrequiredtocompletesurveyquestions.TheQualityCultureSurvey,estimatedtotakeapproximatelyfiveminutesperrespondent,wascompletedbymorethan10,000staffinWave1.Europeansurveysrequiredapprovalbyunionrepresentatives;thiswasalsonotincludedintheestimationofeffort.
Inaddition,Wave1Pilotdatacollectionandsubmissionsiteswereallowedtoprovide“goodenough”data.ThismeansthattheymaynothaveconductedallthecheckingandapprovalstepsthatwouldotherwiseberequiredforformalsubmissiontoFDA.
McKinseyestimatedtheoperationaleffortrequiredtosetupandsupporttheWave1Pilot, including:
f Preparing submission templates f Establishing the database f Definingthecollectionprocessfordatasubmission f Supporting companies in submitting their data f Analyzing the data
TheseestimatesdidnotincludethetimespentbyMcKinseypersonnelworkingaspartofISPE’sprojectteam,contributingtothepilotdesignanddefinitionsdevelopment,producing the report for ISPE and individual companies’ benchmarking reports.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 21
3
3.6 Data Collection Period
Part of the announcement at the launch of the Wave 1 Pilot included details of the data collection period:
Companies to provide data [emphasis added] for approximately one year (historic) and 3-months “real-time,” but individual flexibility possible to accommodate data availability.
AprimarygoaloftheWave1Pilotwastohavefindingsaccruedbytheendof2014sothattheycouldbeavailableeitherbeforeFDAissueditsFederalRegisterqualitymetricnotificationorforconsiderationduringthepubliccommentperiod.Giventhistighttimeline,andtoensuremeaningfuldatawascollectedinshortorder,itwasdecidedtocollectdatausingtwodataperiods–retrospectivelyfor12monthsforcertainmetricswherecompanydataalreadyexisted,andprospectivelyfor3monthsfor metrics that may not have been previously collected or measured at a site. Templatesweredesignedtocollectdata.AnexampleofsitedatafrequencyandcollectionperiodisgiveninTable1,showinganonsterilefinisheddosageformasan example.
Table 1: An Example of Frequency and Period of Collecting Retrospective and Current (Prospective) Data in the Pilot
Data Points/MetricRetrospective (Nominal)
Current (Nominal)
Baseline Data
Production volume in units
Monthly for 12 months Monthly for 3 months
Packs released
Lots dispositioned
Lotstested—total
Lotstested—stabilityonly
Site head count
Annual 3 MonthsSitequalityheadcount
Number of products
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 22
3
Data Points/MetricRetrospective (Nominal)
Current (Nominal)
Site Data
Rejectedlots
Monthly for 12 months Monthly for 3 months
Reworked/Reprocessedlots
ConfirmedOOS—total
ConfirmedOOS—stabilityfailuresonly
UnconfirmedOOS
Total recall events
RecallEvents—ClassIandII
Rejectedlots
Total recalled lots
Total complaints
Critical complaints
ProductssubjecttoAPQRAnnual No data collected
APQRontime
NumberofCAPAswitheffectivenesschecks
3 monthly in 4 periods, April 2013 to March 2014 One 3 month period
NumberofeffectiveCAPAs
Number of deviations
Number of recurring deviations
Product Data
Total complaints for the product for the reporting year
Annual for individual products
Current period, typically 3 months for individual products
Total critical complaints for the product for the reporting year
Total packs released for the product for the reporting year
Total lots dispositioned for reporting year
Total lots tested for reporting year
Rejectedlots
ConfirmedOOS
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 23
4 Operational Processes for the Wave 1 Pilot
This section discusses some of the key operational aspects of the Wave 1 Pilot for both the McKinsey support team and the participant companies.
4.1 McKinsey Operational Process
TheMcKinseysupportprocessesforperformingtheworkassociatedwiththeWave1PilotwerebasedontheirexperiencegainedfromtheirPOBOSbenchmarking programs.
Anoverviewoftheprocessisasfollows:
f Preparing templates for data submission: – Differenttemplateswererequiredfordrugproducts,sterileandnonsterile
drug products, and for labs. – Templatesforeachmetricincludeddetaileddefinitions,fieldsforeachdatapointtimeperiod–monthly,quarterlyorannual–andacommentaryfield.
– Templateswerevalidatedusingbuilt-inchecksfordataconsistency(e.g.,totalOOSbyproduct=totalOOSforsite)andlockedbeforetheyweresentto sites.
f Setting up databases for input and analysis. f Definingthedata-collectionprocess. f Translatingthequalityculturesurveyintotheappropriatelanguageforeach
participating site. f Answeringexploratoryquestionsfrominterestedcompanies,suchas:
– Howmuchtimeandeffortandresourceswillweneedforthepilot? – Howmuchdoesitcost? – Whatisinvolved?
f Enrolling companies into the Wave 1 Pilot by arranging: – Confidentialityagreements – Purchasing orders – Explainingdatasubmissionrequirements
f Answeringquestionsduringthedata-collectionphaseandupdatingthe FAQs document.
f Reviewingandclarifyingdata(i.e.,potentialoutliers). f Processing the Quality Culture and Process Capability Surveys. f Analyzingdata,runningcorrelations,profilingmetrics. f ReportingresultsoftheWave1Pilotdataanalysis.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 24
4
Usingagreed-upondefinitionsandsurveyquestions,projecttimelinesanddata-collectionfrequencies,McKinseypreparedthetemplatesinExcel.Anexampleisprovided in Appendix 3.
For the Wave 1 Pilot, companies completed the Excel spreadsheets manually and sent them to the McKinsey support team. An automated data-entry process may be considered for the future.
AdatalockwasappliedattheendofNovember2014,andallparticipatingcompanies complied.
4.2 Experiences from Participating Companies
Companies participating in the Wave 1 Pilot provided feedback throughout the data collectionandanalysisphases;PilotLeadsmeetingsheldbytheIndustryEngagementSubteam also provided feedback.
ThemajorityofcompaniesthatparticipatedintheWave1Pilotreportedbenefitsarisingfrom their involvement. These included:
f The opportunity to trial a centralized submissions process gave them a deeper understandingoftheimpactofstandardizedmetricsdefinitionsanddesign.
f Participation enhanced the maturity of their internal metrics programs. f Eachparticipatingsitereceivedaconfidentialbenchmarkingreportthatoutlinedtheirperformancewithrespecttotheirpeergroup(s).
Withrespecttoreporting“goodenough”data,someparticipantsnotedthatsomedatatheycollectedwerederivedfromnon-GMPsystems(e.g.,productportfolios).Discussions have indicated the need for a validated/cGMP-based metrics collection, storageandreportingsystemthatcouldbereviewedbyinspectionteamstoconfirmthe veracity of any metrics reported to FDA. Concerns raised about the potential burden associatedwiththiswillrequirefurtherconsideration.
A detailed example of some of the key aspects of one company’s experience of participating in the Wave 1 Pilot is provided in a case study in Appendix 4. A summary of the case study’s main points are:
f CompanyAhasalargeproductrangeandverycomplexsupplychains,whichmakeassigningproduct-levelmetricsextremelydifficultandtime-consuming.
f Changing their current IT systems to a standardized set of metrics that could produceproduct-leveldatawouldrequiresignificantinvestment.
f DatareportedintotheWave1Pilotwere“goodenough”toexaminethedata-collectionand-submissionsystems’,mechanics,buttheywerenotsubjectedtothereviewandcheckingthatwouldberequiredforofficialsubmissiontoFDA.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 25
5 Findings from the ISPE Quality Metrics Initiative Wave 1 Pilot
ThissectiondiscussesthemainfindingsfromtheISPEQualityMetricsWave1Pilotand includes considerations of the sample size, metric and survey data analysis, collectionandsubmissioneffortdataandthekeyrelationshipsobserved.
5.1 Sample Size
The Wave 1 Pilot collected data from 18 participating companies at 44 individual sites. Distribution of the sites by technology, type of product/business, region and company size is given in Figure 6 and Table 2.
Figure 6: Sample Distribution of Participating SitesDiverseSample:18ParticipatingCompanieswith44Sites/Technologies
468 8
18
18
Other API Bio DS Sterile Solids
567
26
CMO, labs
Cons. health
Gx Rx
73
16
LA EMEA NA Asia
By technology By type of product
By region By company size
39
5
Large Small
Note:Ifasitehasmorethanonetechnologywecountthenumberofseparatetemplatestheywillfill,usuallyonepertechnology
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 26
5
Table 2: Figure 6 Abbreviations
Technology
Bio DS Biopharmaceutical or biological drug substance site
API Small-molecule drug substance (active pharmaceutical ingredient)
Type of product
Rx Originator company
Gx Generic company
Cons. health Consumer health or OTC
CMO Contract manufacturing organization
Labs Contract research and testing laboratories
Region
NA North America
EMEA Europe, Middle East and Africa
LA Latin America
Company size
Small < $1 billion in revenues
Large > $1 billion in revenues
The Quality Culture Survey sample size comprised 10,300 respondents from 37participatingsites.Thisdiffersfromthetotalpilotsamplesizeof44sitesbecausesomesiteshadtwodifferentproducttechnologieslocatedonthesamephysicalsite.ThesesitescompletedtwoWave1Pilotmetricstemplatesastwoseparatesites,yetsubmittedtheirqualityculturesurveyassessmentsasonesite.
TheWave1Pilotsamplewasconsideredsuitablydiverse,bothbyregionandtechnology, to facilitate a representative analysis. Other aspects of the sample, however,hadsimilaritiesthatshouldbeacknowledged:
f Most participant companies originated from developed countries, and therefore didnothaveanysignificantlanguageorinterpretationissueswiththestandardizeddefinitionsorthepilottemplatesubmissioninstructions.
f Allenrolledsitesmaybeconsideredingoodstandingwithrespecttoquality;theydidnothaveanyqualityorcompliance(e.g.,consent)issues.
f Themajorityofcompanieswereclassifiedaslarge. f All companies and their sites consented to enroll in the Wave 1 Pilot and, therefore,hadanopenandpositivedispositiontotheuseofqualitymetricstomonitoranddriveenhancedqualityperformance.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 27
5
5.2 Metric and Survey Data Analysis and Discussion
Toassistwithdataanalysisandpresentation,metricscollectedinthepilotweregroupedintodifferentcategoriesshowninTable3.Theseinclude:
f External Quality Outcomes:Anoutcomethatmayaffectthepatientdirectly(e.g., the patient makes the complaint) or indirectly (e.g., a recall leads to product unavailability).
f Internal Quality Outcomes: An outcome observed by a company that could affectbusinessoutput(e.g.,arejectedproduct),productdoesnotleavethecompanycontrol,however,andisnotavailabletothepatient.
f Supplier Quality: ConfirmedOOSofanincomingrawmaterialisameasureofsupplierquality.
f Laboratory Quality: AnunconfirmedOOSisameasureoflaboratoryquality,whetherornotthecauseisidentified.
f Site Maturity: Metrics that could be considered measures of site maturity, suchasannualproductqualityreviews(APQRs)completedontime,highvaluesofcorrectiveandpreventiveaction(CAPA)effectivenessrateandlowvaluesofrecurring deviations rate.
Table 3: Categories of Metrics
Categories Metrics
External Quality Outcomes (market)
Totalrecallevents—US
RecalleventsClassIandII—US
Recalledlots—US
Total complaints rate
Critical complaints rate
Internal Quality Outcomes
Lot acceptance rate
ConfirmedOOSrate—release
ConfirmedOOSrate—stability
Deviations rate
Rightfirsttime(rework/reprocessing)rate
Mediafillssuccessful
Environmental monitoring (EM) action limit investigations rate
EM action limit rejects rate
Supplier Quality ConfirmedOOSrate—incomingmaterials
Laboratory Quality
Site Maturity
UnconfirmedOOSrate
APQRcompletedontime
CAPAseffectiverate
Recurringdeviationsrate
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 28
5
5.3 Wave 1 Pilot Quality Metric Data Analysis
Individualcompanydatawascomparedwiththetotalsampletodevelopabenchmarkingreportforeachcompany.Thesedatawereconfidentialtothecompany and McKinsey. ISPE did not have access to individual company data.
Total industry-level data for all metrics collected, relationship determinations and comments are provided in Appendix 5 of this report.
Totalindustry-leveldataforeachmetricweretestedandevaluatedasappropriate,using either scatter plots (correlation of a leading indicator vs. an outcome) or quartilesanalysis(rangeofoutcomevaluesagainstleadingindicatorvaluessplitintoquartiles)orprofiling(formetricswithdiscretevalues).
ManyfigurespresentedinAppendix 5 of this report also contain a summary ofthestatisticaltoolsappliedandincludeexplanation,wherenecessary.
Asperstandardstatisticalpractices,incompletedataandextremeoutlierswereexcludedfromanalyses.Theresultingsamplesizesallowedstatisticalanalysiswithsomelimitations,asfollows:
f Toallowforsufficientsamplesize,mostanalysesweredoneforfinisheddosagesites overall, not by technology.
f Productdatawascollectedonannualbases,anddidnotallowtime-laganalysistoseehowproductmetricscorrelateovertime.
f Anyrelationshipsidentifiedbetweenmetricswerestatisticallysignificant(lessthan5%likelihoodofacoincidence),however: – Thestrengthoftherelationshipsvaryandmayberelativelylow (e.g.,somemaycorrelatewithR2of30%or40%),sincethesemetricsareinfluencedbymultiplefactors.
– Correlation doesn’t imply causation. Understanding the underlying factors anddirectionofarelationshipwillrequirefurtherwork,ideallyonalargerdata set from a larger sample size.
From the total industry database, median ranges of individual metrics split bytechnologyaregiveninFigure7andFigure8below.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 29
5
Figure 7: Metric Ranges by Technology: Solid Dosage and Sterile
External (market) quality outcomes
Internalquality outcomes
Supplier quality
Lab quality
Site maturity
� Recall events – US� Recall events class I and II – US� Recalled lots – US� Complaints rate (per million packs)� Critical complaints rate (per million packs)
� APQR on time (%)� CAPAs effective (%)� Recurring deviations (%)
� Lot acceptance rate (% released out of all finally dispositioned lots)� Confirmed OOS – release (per 000’ lots release-tested)� Confirmed OOS – stability (per 000’ stability lots tested)� Deviations rate (per 000’ lots dispositioned)� Rework rate (% lots dispositioned)� Media fills successful (%)� EM action limit investigations rate (%)� EM action limit rejects rate (%)
� Confirmed OOS – incoming materials (per 000’ RM/PM lots tested)
� Unconfirmed OOS (per 000’ lots tested)
Solids Steriles
0.00.00.0
0.00.00.0
0.00.00.0
60.2
16 240.4
1.00.51.5261.2 0
0.00.00.0550.1
0.00.00.0761.7
99.7% 99.4% 98.8% 99.6% 98.8% 96.1%0.8 1.5 2.9
0.0 2.3 6.20.5 1.0 8.3
0.0 2.5 6.3257 472 74494 134 230
0.0% 0.3% 0.7%0% 0.1% 1.0%
0.9 2.7 5.40.3 1.1 4.9
0.6 0.3 4.11.6 3.2 4.3
90.9%100% 100% 100% 100%96.2%
96.7% 94.9% 92.6%98.5% 97.4% 33.4%5.0% 9.5% 23.6%5.2% 12.9% 27.5%
100% 100% 100%0.6% 2.12% 8.8%
0 0 0.02%
TQ BQMedian TQ BQMedian
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 30
5
Figure 8: Metric Ranges by Technology: API, Bio and OtherMedian
0.00.00.0
0.00.00.0
N/A N/AN/AN/A
0.00.00.01110.9
100% 95.3% 98.1%
2.9 25.1 6.10.9 15.4 0.0464 9678 154
1.6% 0.2% 0.6%
3.6 2.2 3.7
4.2 1.7 1.4
100% 87.5% 100%95.8% 33.2% 64.9%7.5% 13.6% 20.7%
API Other FPBio
External (market) quality outcomes
Internalquality outcomes
Supplier quality
Lab quality
Site maturity
� Recall events – US� Recall events class I and II – US� Recalled lots – US� Complaints rate (per million packs)� Critical complaints rate (per million packs)
� APQR on time (%)� CAPAs effective (%)� Recurring deviations (%)
� Lot acceptance rate (% released out of all finally dispositioned lots)� Confirmed OOS – release (per 000’ lots release-tested)� Confirmed OOS – stability (per 000’ stability lots tested)� Deviations rate (per 000’ lots dispositioned)� Rework rate (% lots dispositioned)� Media fills successful (%)� EM action limit investigations rate (%)� EM action limit rejects rate (%)
� Confirmed OOS – incoming materials (per 000’ RM/PM lots tested)
� Unconfirmed OOS (per 000’ lots tested)
AlthoughafewsitesexperiencedarecallduringtheWave1Pilotperiod,themedianforrecallsinbothFigure7andFigure8waszero.(Forfurtherdetailssee“RecallEvents” in Appendix 5.)
In addition to the data collected, participating sites provided very useful feedback on thedefinitionsusedinthepilot.Anexampleofthisincludestheextensivefeedbackreceivedonthedefinitionfor“recurringdeviationsrate,”showninFigure9.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 31
5
Figure 9: Feedback on Definition of Recurring Deviations Rate
Pilot definition
Numberofdeviations(outofthetotalreportedinline49)forwhichduringthe12monthperiodprecedingeachdeviation,atleastoneotherdeviationhasoccurredwiththesamerootcausewithinthesameprocessand/orworkarea.
Feedback/ alternative definitions
f Periodconsideredfortherecurrencemaybebasedonthetypeofissueorworkarea(e.g.dependsontheoccurrenceofthespecificprocesswithdeviation),orlefttothequalitypersonneljudgment;
f Somesitesuse6monthsor2yearsaslookbackperiod; f Deviationmaybeconsideredrecurringifreoccurredanywhereintheplant,notjustinthesameworkarea f AtleastonesiteconsidersrecurringonlydeviationsthathavehadaCAPA(asrecurrenceindicatesineffectiveCAPAimplementation(otherdeviationswithsamerootcauseareconsidered“repeat”))
f Manysitesfeelthatfinalassessmentdependshowdeepyougointotherootcause–fromamoregeneral“operatorerror”toaveryspecificerrordescription(hastobethesameproduct,natureofincident,rootcausecategory)–whichwouldinfluencehowrecurrenceisidentified;
Recurringdeviationsratewasnottheonlymetricthatproducedlotsofquestionsregardingdefinitionandhowthedatashouldbecalculatedandsubmitted.Otherdefinitionsinneedofrefinementandfurtheralignmentarecriticalcomplaints,reworkrateandCAPAeffectivenessrate.
5.4 Wave 1 Pilot Quality Culture Survey Data Analysis
Forthequalityculturesurvey,eachofthe15questionscouldbeansweredusingoneoffiveresponseoptions:
f Strongly agree f Agree f Disagree f Strongly disagree f Ican’tanswerthisquestion
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 32
5
Tofacilitatedataanalysisandrelationshipmappingwiththequalitymetricsdataset,scoringmechanismwasestablishedbasedonthe“topboxes”approach.Foreachquestion,theproportionof“Stronglyagree”and“Agree”answerswascalculated.Topboxesanalysisassignsa1(or100%)ifallrespondentsreply“Stronglyagree”or“Agree,”and0(0%)ifallrespondentsreply“Disagree”or“Stronglydisagree.”Theseproportionalvalueswerethenplottedonaradardiagramforeachquestion,asshowninFigure10.
Figure 10: Quantitative Quality Culture Scores Plotted on a Radar DiagramTotalof10,300respondentsfrom37sites
0.6
0.7
0.8
0.9
1
Patientfocus
Training
Problem solving
Recognition
Metrics
Knowledge
Continual improvement
Coaching Dialogue
Gemba
Awareness
Responsibility
Openess
Ethics
Motivation
Score1
Capabilities
Governance
Leadership
Mindset
Integrity
1 Totalscorecalculatedas“topboxes”(shareof“agree”and“stronglyagree”responses)ratio.100%=allrespondentsagreeorstronglyagree,0%=nobodyagrees or strongly agrees.
Theresultsshowedthatatthesitelevel,questionspertainingtotheculturalelementsofCapabilitiesandIntegrityreceivedthehighestratings,whilethoseassociatedwithGovernanceandLeadershipscoredrelativelyweaker.Thetop-boxesapproachfacilitatedaquantitativeanalysisofthequalityculturefindingsandprovidedsomeinterestinginsights,butitisbroadlyagreedthatfurtherworkisrequired,potentiallyinaWave2Pilot,tounderstandthesefindingsbetter.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 33
5
Therangeofvaluesassignedtotheresponsesforeachqualityculturequestionaregiven in Figure 11. Orange shading represents the range of site responses that fall belowtheindustryaverage,whilelightorangeshadingindicatestherangeofsiteresponses exceeding the industry average.
Figure 11: Range of Values for Quality Culture Responses
0.05 0.15 1.00 0.50 0 0.80 0.90 0.45 0.25 0.35 0.55 0.65 0.10 0.20 0.30 0.40 0.60 0.70 0.75 0.85 0.95
Maximum Minimum AverageSurvey Question Avg. And ranges
Min
dset
s In
tegr
ity
Lead
ersh
ip
Gov
erna
nce
Cap
abili
ties
Training: The training I have received clearly helps me to ensure quality in the end product
Patient focus: I know which parameters of our products are particularly important for patients
Problem solving: All line workers are regularly involved in problem solving, troubleshooting and investigations Recognition: We recognize and celebrate both individual and group achievements in quality Metrics: Up-to-date quality metrics (e.g. defects, rejects, complaints) are posted and easily visible near each production line Knowledge: Each line worker can explain what line quality information is tracked and why Continual improvement: We are regularly tracking variations in process parameters and using them to improve the processes Coaching: Supervisors provide regular and sufficient support and coaching to line workers to help them improve quality Dialogue: We have daily quality metrics reviews and quality issues discussions on the shop floor Gemba: Management is on the floor several times a day both for planned meetings and also to observe and contribute to the daily activities Awareness: Every line worker is aware of the biggest quality issues on their line and what is being done about them Responsibility: All employees see quality and compliance as their personal responsibility Openness: I am not afraid to bring quality issues to the management’s attention Ethics: People I work with do not exploit to their advantage inconsistencies or “grey areas” in procedures
Motivation: All employees care about doing a good job and go the extra mile to ensure quality
Responsestoquestionsonethics,knowledge(e.g.,metricstracking),andGemba(Japanesefor“atthesite,”itrefersobservationsofaprocessinaction,ormanagementpresenceontheshopfloor)hadthehighestvariationinresponses.
Toexplorewhethertherearedifferencesinresponsestoquestionswithineachoftheculturaldimensionexamined,thevaluesforeachdimensionwereplottedagainstvaluesoftotalqualitycultureresponses.TheseplotsaregiveninFigure12.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 34
5
Figure 12: Plot of Total Quality Culture Values for Each Quality Culture Dimension
00.20.40.60.81.01.2
0.4 0.6 0.8 1.0 1.2
QTotal
QLeadership
00.20.40.60.81.01.2
0.6 0.8 1.0 1.2
QTotal
QMindset
00.20.40.60.81.01.2
0.6 0.8 1.0 1.2
QTotal
QIntegrity
00.20.40.60.81.01.2
0.7 0.8 0.9 1.0 1.1 1.2
QTotal
QCapabilities
00.20.40.60.81.01.2
0.4 0.6 0.8 1.0 1.2
QTotal
QGovernance
Leadership
Capabilities Integrity
Governance Mindset
R2=0.9974
R2=0.8935
R2=0.8192
R2=0.9283
R2=0.8781
1 Totalscorecalculatedas“topboxes”(shareof“agree”and“stronglyagree”responses)ratio
Thisanalysisshowsthattheculturaldimensionsincludedinthequalityculturesurveyarehighlyconsistentbetweeneachother—i.e.,thelineslopesaresimilarandeachdimensionishighlycorrelatedwiththeoverallvalue.
Thisanalysisconfirmedthatattemptstodeterminerelationshipsbetweenqualitycultureandotherqualitymetricsvaluescanusetotalqualityculturevalues.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 35
5
5.5 Data Collection and Submission Effort Data
Estimatesofeffortexpendedbycompaniesasanindustrytotalwereprovidedandthe results of this analysis are provided in Appendix 5 for each individual metrics.
Theaveragetimeoneachsitetocollecttheannualdatawas88hours.Effortestimates(hours)splitbetweenthevarioustypesofbusinessarealsogiveninFigure13below.
Figure 13: Average Time to Collect Annual Quality Metric DataTotaltimespentoncollectingdata(12months),[Hours]
54
156 162
80
14
88
434
499
434422
55
120
14 138 84
Average (minimum and maximum as range)
Drug substance
Gx
Total¹
OTC
Rx
Labs
28
1 Excluding DS and Labs
TheresultsshowthatOTCandgenericsitestookthreetimeslongerthanoriginatorcompaniestocollectsimilardata.Participantsseemedtoindicatethisdifferencecould be due to issues such as increased number of products and (potentially) the increased complexity of supply chain for OTC and generic sites. Site volumes (packs or lots dispositioned) and site complexity (number of products) did not appear to influencethecollectionandsubmissioneffortrequired.
Drugsubstanceandlaboratorysitesrequiredsignificantlylesstimethanfinishedproduct sites because less data is collected. Even though the Wave 1 Pilot had limitedvisibilityinsuchsites,theirworkloadwascalculatedat45hourscollectionandreportingpersite,theaverageworkloadforbothlabanddrugsubstancesites.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 36
5
Approximately12,000sitesgloballyhaveFederalEstablishmentIdentifier(FEI)numbers;[17] close to 6,000 are registered as Finished Dosage and API sites, and the remaining 6,000 sites include medical gases, medical feed, labs, Center for Biologics Evaluation andResearch(CBER)establishments,andothers.Wave1PilotdataestimatedtheaverageworkloadfortheAPIandFinishedDosagesitesat90hours,whiletheaverageworkloadfortheothersitesisestimatedat45hours.Usingatypicallaborcost(includingoverhead)of$75,000peryear,collectingthisamountofdatawouldcosttheindustryapproximately an additional $35 million annually.
Thisestimateisconsideredconservative,however,becauseitdoesnotincludeseveral factors, such as:
f Wave1Pilotsiteswereallowedtoprovide“goodenough”data.SubmissiontoFDAcouldrequiremorethoroughandcompletedatacollection,additionalreviewanddataverificationsteps,potentiallyatdifferentlevelsanddisciplinesandwouldhavetobeaccompaniedbyconsideredcomments.
f Timeforinternaldiscussions,managementreviewandabove-siteguidancewasnotincluded.
f Efforttodevelopandvalidatenew/modifiedITsystemswasnotincluded.(ThiswasnotrequiredfortheWave1Pilot.)
f Participantshadflexibilitytoprovidemostpragmaticdataset (e.g., for all products at site or only those for the US market).
f Datawereprovidedwithineachsite,notthroughfullproductsupplychain. f Participants had mature systems and capabilities. f Majorityofsiteswerefromdevelopedcountries.
Theadditionalcosttoproduceofficialsubmissionscouldbringtheannualcostof such a program to $100+ million.
The time spent to collect the 3 months of data for the three metrics collected at both productandsitebases(lotacceptancerate,confirmedOOSandcomplaints)aregivenin Figure 14.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 37
5
Figure 14: Time Spent To Collect Product Level Data and Site Level Data for 3 MonthsTime spent on collecting site1andproductleveldata,[Hours,(15months)]
19.4
76.0
9.6
OTC Rx
Drug substance
Metrics collected at both site and product level:
Lot acceptance rate Confirmed OOS Complaints, ppm
17.5
87.0
5.0
Site level data1 – Hours
Product level data – Hours
1.9 -11 4.6
xx Difference between site and product data time
Note:Sitesthatdidn’tsubmitfullproductdatawereexcludedaseffortlikelytobeunderstated.Gxsamplewastoosmalltoreportresults1 Onlyformetricswithproductlevelgranularity
NodataisincludedintheanalysisshowninFigure14forGenericcompaniesduetoinsufficientsamplesize.Alsositesthatdidnotsubmitfullproductdatawereexcludedsincetheywerelikelytounderestimatetheeffortrequired.
ThefindingsfromtheanalysisshowninFigure14are:
f Rxsiteswereabletocollectthesemetricsatproductlevelaseasilyasatsitelevel(therewasanapproximately10%differenceintimeforthethreemetrics)due to several factors.
f SomeOTCsitesfoundproduct-leveldataforthesemetricsmoredifficulttocollectthansitedata(15%moretimeforthethreemetrics).
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 38
5
PotentialreasonswhyRxsiteswereabletocollectproductleveldataaseasilyas site level data include:
f Thesethreemetricswerechosenintentionallytobecollectedeasilybyproductlevel. f Some sites already had systems set up to collect product-level data. f FortheWave1Pilot,productdatawascollectedwithineachindividualsite
rather than across a full supply chain. f Mostsitesselectedforthepilothadgoodsystemsandproficientpersonnel;
some even had systems set up to allocate data by product and then aggregate it at site level.
f Definitionsallowedaccurateallocationbyproduct(e.g.,usinglotsdispositionedrather than lots attempted).
OTCsitesspentmoreeffortcollectingsitelevelmetricsfromFigure13andFigure14,andevenmoreeffortcollectingproductlevelmetricsfromFigure14,potentiallydueto:
f Separatingcomplaintsdowntoindividualformulationsratherthanataproductfamily(e.g.,severalflavorsorcolors)level.
f Countingpackorunitdata(vs.cases),anddisaggregatingdatafromAPR’sto match the time frame of the data collection period.
Using the full 15 months of data, the time to collect individual metrics is listed inFigure15,withcolumnsgivenforRx,GxandOTCcompanies.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 39
5
Figure 15: Effort per Metric for Rx, Gx and OTC Companies15 months of data
Rx Gx OTC
Time [hours]
Thisanalysisindicatesthatthemosttime-consumingmetricstocollectwere;OOSforfinalproductandcomplaints,bothtotalandcritical.Theleasttime-consumingmetricstocollectwererecallsandAPQRsontime.
Inadditiontocapturingthenumberofhoursrequiredtocollecteachmetric,Wave1Pilotcompanieswerealsoaskedtoreportthedegreeofdifficultyofcollectingaparticularmetricusingascaleof1to4,(1beingeasiestand4mostdifficult).ThisanalysisisshowninFigure16,asestimatedbytheparticipatingcompaniesthemselves.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 40
5
Figure 16: Degree of Difficulty of Collecting Each Metric
Rx Gx OTC
Difficulty[1(easy)-4(difficult)scale
1-Veryeasy,allthedatawehavein an electronic system/paper based documents in the form requiredhere
2 - Fairly easy, most of the data wehadready,somewehadtorecalculate/aggregate basing ontheprovideddefinitionsandourrawnumbers
3-Difficult-wehadtorecalcu-late/aggregate most of the data, sometimes from very fragmented sources
4-Verydifficult,wedidnothavethe data at all for retrospective period,weonlystartedcollec-ting it for current period
Average
3 quartile 1 quartile
Thisanalysisindicatesthatthemostdifficultmetricstocollectweretherecurringdeviationsrateandthetwosterilespecificmetrics.TheleastdifficulttocollectwereonceagainUSrecallsandAPQRsontime.
Figure17showsafurtheranalysisrelatingtoeaseofdataaccessandburdentocollect,plottingtheaveragedifficultyratingagainstthemediantimefordatacollectionforRxcompanies.Thisgroupwaschosenastheypresentedthelargestsamplesizein the Wave 1 Pilot.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 41
5
Figure 17: Average Time for Collection of a Metric and Median Time for CollectionsFinisheddosageRxsample,15monthsofdata
1
2
3
0 5 10 15 20
Lot acceptance
Investigations related to action limit excursions
Confirmed OOS - product
CAPA effectiveness
APQRs on time
Media fill successful
Median time for data collection [hours]
Recalled lots (US)
Confirmed OOS - stability Rework rate
Average difficulty rating
Recurring deviations
Total complaints rate
Confirmed OOS - RM
Recall events - class I and II (US)
Unconfirmed OOS rate Lot rejects related
to action limit excursions
Critical complaints rate
Deviations rate Recall events (US)
TheresultsfromFigure17showthatthemosttime-consumingmetricsarenotnecessarilyratedasthemostdifficult.
Theredcircleshighlightthemostdifficulttocollectmetrics:
f Investigations and lots rejected related to environmental monitoring for sterile products.
f Recurringdeviations:Themajorityofsitesreportedthattheydonotcurrentlyhave processes in place to collect this metric accurately. Where it is collected thereisbroadvariationinthedefinitionof“recurrence.”
AsshowninFigure15,OOSforproduct,andcomplaints,totalandcriticalwerethe most time-consuming metrics to collect.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 42
5
5.6 Process Capability Survey
The results of the Process Capability Survey (see Appendix 2) are given in Figures 18, 19and20.Responsestothefirsttwohigh-levelquestionsinaregiveninFigure18.
Figure 18: Response to Process Capability Questions
No 5
Yes 95
Do you measure that the process remains in a state of control during manufacturing?
To what % of products is it applied? 1
26
431
4
30% - 40%
10% - 20%
70% - 80%
90% - 100%
0% - 10%
1 Whenonlysomeproductswerechosen,choicewasbasedonriskapproachtocustomerandimportanceforbusiness
FromFigure18weobservethat95%ofsitesapplyongoingmonitoringduringproductionprocessestoanaverage74%oftheirproducts.Whereprocessmonitoringwasappliedtoaselectedrangeofproducts,arisk-basedapproachwastypicallyusedtodeterminewhichproductsrequiredmonitoring.
Moredetailedquestionsaskedwhichprocesscapabilitystatisticaltoolwasmostusedandtowhatattributes[e.g.,criticalqualityattributes(CQAs)]orparameters[e.g.,criticalprocessparameters(CPPs)orin-processcontrols(IPCs)].ResponsestothesequestionsaregiveninFigure19andFigure20.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 43
5
Figure 19: Process Capability Survey FindingsPercentage of sites using each type of metric1
12%
46%
5%19%19%21%
Other2
21%
Trending
42%
Box plots
0%
Tolerance interval
19%
PpK
14%
CpK
16%Applied to CQA
Applied to CPP
CQA
IPC
CPP
28%
77%
7%23%
35%39%
1 Outofsitesmonitoringcapabilityinanyway2 Othermentionedmetricswereparetocharts,monitoringviaexcursionstrending,I-chartsregression,3sigma,andRun/controlcharts
Theresultsindicatethattrendingismostthewidelyusedtool.Processcapabilityindex(CpK),processperformanceindex(PpK)andtoleranceintervalsareusedlessoften—by39%and22%ofsitesrespectively.While,asseeninFigure20,91%ofsitesmeasuretheircurrentstateofcontrolthroughCQAs,whileonly56%onIPCsand61%onCPPs.
Figure 20: Process Capability Tools in UsePercentage of sites monitoring each parameter type
CQA IPC CPP
91%
56% 61%
As anticipated the capability approach is variable across companies and in terms of use and applicability. The tool employed e.g. Ppk, Cpk, control charts etc. is contextual and there is no one tool that can be applied to all situations. These tools should be used internally for troubleshooting and identifying continual improvement opportunities rather than for monitoring compliance.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 44
5
5.7 Establishing Statistically Significant Relationships
RelationshipsbetweenthecollectedmetricsweretestedandassessedusingthegroupingofquantitativemetricsshowninTable3(externalqualityoutcomes,internalqualityoutcomesandsitecultureandmaturity).ConnectionsbetweeneachofthestandardizedqualitymetricsandQualityCultureSurveyvaluesaredepictedbytheorange lines in Figure 21.
Figure 21: Relationship Testing
Total complaints
Critical complaints US recalls
External (market) quality outcomes
Deviations recurrence Culture APQR on
time CAPA
effective
Site culture and maturity (leading indicators)
Internal quality outcomes
Relationship testing – annual and quarterly data, overall and by technology
Lot acceptance
Deviations rate
Action limits excursions
(sterile)
Confirmed OOS
Invalidated OOS
Stability failures Rework
Asbefore,incompletedataandextremeoutlierswereexcluded.Theresultingsamplesizesallowedstatisticalanalysiswithsomelimitations:
f Toallowsufficientsamplesizemostanalyseswereperformedforfinisheddosage sites overall, not by technology
f Productdatawerecollectedonannualbasis,notallowingtimelaganalysistoseehowproductmetricscorrelateovertime
Identifiedrelationshipsbetweenmetricsweredeemedstatisticallysignificantwhentherewasalessthan5%likelihoodofacoincidence.Thestrengthoftherelationshipsmayvary,however,andinsomecasesarerelativelylow(e.g.,somemaycorrelatewithR2of0.30(30%)or0.40(40%).ThesizeoftheWave1Pilotdatasetisacknowledgedinthesefindingsanditisalsonotedthatthesemetricsunderexaminationareinfluencedby multiple factors not currently included in this analysis.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 45
5
R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationship)to1(perfectlinearrelationship).Pearsoncoefficient(R)measurestheextenttowhichtwovariablesmoveinthesamedirection.Itvariesfrom 0 (random relationship) to 1 (perfect linear relationship).
TherelationshipsdeterminedfollowingthisanalysisaresummarizedinFigure22.
Figure 22: Significant Relationships
Total complaints
Critical complaints US recalls
External (market) quality outcomes
Deviations recurrence
Culture
APQR on time
CAPA effective
Site culture
Lot acceptance
Deviations rate
Action limits
excursions (sterile)
Confirmed OOS (release)
Invalidated OOS
Stability failures Rework
Internal quality outcomes
Site maturity (leading indicators)
Demonstrated relationship within the Wave 1 sample Statistically significant Significance >0.05
Theorangelinesshowrelationshipsthatarestatisticallysignificant(<0.05%),whilethegreylineshowsrelationshipsthathaveasignificancelevelexceeding0.05%butarestillconsideredworthexaminingfurthergiventhelevelofvariabilityandrelativelylowsamplesize.
Multivariatecorrelationanalysiswasnotperformedsincethesamplesizewasinsufficientforthis.
A statistically significant relationship does not imply causation. Causation can only be proposed after studying underlying factors, which requires further work, as does understanding the degree of correlation and direction of a relationship.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 46
5
5.8 Statistically Significant Relationships in Wave 1 Pilot Data
Statisticallysignificantrelationshipswerefoundbetweenthefollowingmetrics,orqualityculturevalues,andthosemetricsasshownintheappropriateanalysisfigure.
f Critical complaints and deviations rate (Figure 23). f US recalls and deviations recurrence (Figure 24). f Lotacceptancerateandrework(Figure25). f Lotacceptancerateandqualityculturevalues(Figure26). f Lotacceptancerateanddeviationsrecurrence(Figure27). f Actionlimitexcursions(sterileproducts)andqualityculturevalues(Figure28). f APQRontimeandCAPAeffectivenessrate(Figure29).
Figure 23: Critical Complaints and Deviations RateCritical complaints in selected intervals of deviations rateSampleof21plants,finisheddosage,averageannualvalues
Median level of critical complaints for sites with higher and lower deviations rate ppm
8.13
1.340.880.34
<0.16 <0.12 <0.39 >0.39 Deviations rate Per dispositioned lot
Sample size 5 6 6 5
Significance: 0.045
Significancelevelisaresultofindependentsamplesmedian(chi-square)test.Valuebelow0.05meansthatthemediansofVariablearesignificantlydifferentbetweencategories.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 47
5
Criticalcomplaintsrateincreaseswithincreaseofdeviationsrate.Valuesfordeviationsratearequartileboundaries.Thecriticalcomplaintsratevaluesarestatisticallydifferent(0.045%)usingthechi-squaredtest.
Figure 24: US Recalls and Deviations RecurrenceRecurringdeviationsforplantswithandwithoutrecallsSample of 35 plants, all technologies, average annual values
Median level of recurring deviations % of closed deviations
38
13
US no recalls US recalls Recalls
Sample size 31 4
Significance: 0.001
+192%
Significancelevelisaresultofindependentsamplesmedian(chi-squared)test.Valuebelow0.05meansthatthemediansofvariablearesignificantlydifferentbetweencategories.
Thereisarelationship(significance0.001%)betweenUSrecallsandrecurringdeviations.Because only four of the 35 sites in this analysis had a recall, causal relationship betweentherecallandrecurringdeviationsratehasnotbeenestablished.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 48
5
Figure 25: Lot Acceptance Rate and ReworkReworkrateinselectedintervalsoflotacceptanceSample of 38 plants, all technologies, average annual values
Median level of rework rate % lots dispositioned
0.62
0
>99 <99 Lot acceptance rate
% lots dispositioned
Sample size 19 19
Significance: 0.001
Significancelevelisaresultofindependentsamplesmedian(chi-square)test.Valuebelow0.05meansthatthemediansofvariablearesignificantlydifferentbetweencategories.Significancerepresentsnon-lineardependency.
Higherreworkrateisassociatedwithhigherlotacceptancerate(fewerrejects)atasignificancelevelof0.001%.Samplesizesaresplitequallybetweentwolevelsofreworkrate.Higherreworkratesarealsoassociatedwithfewerrejects(higherlotacceptancerate),thereforereworkratemayhaveadditionalvalueasbalancingmetric,butthesitepracticesdrivingthisrelationshipstillrequirebetterunderstanding.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 49
5
Figure 26: Lot Acceptance Rate and Quality Culture ValuesSample of 34 plants, all technologies, average annual values
93
94
95
96
97
98
99
100
0.6 0.8 1.0
Lot acceptance rate % of lots dispositioned
Quality culture overall Top 2 boxes
R2 = 0.29 R = 0.54
P-value = 0.001
R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationshipbetweenXandY)to1(perfectlinearrelationship).Pearsoncoefficient(R)isameasuretowhatextenttwovariablesmoveinthesamedirection.Itvariesfrom0(randomrelationship)to1(perfectlinear relationship) or -1 (perfect negative linear relationship). P-value is probability that correlation is zero (in this case this means there is no linear correlation betweenXandYvariables),valuebelow0.05indicatessignificantresults.
Strongerqualityculturevalues(total)areassociatedwithhigherlotacceptancerate.Thesignificancelevelof0.001%isstrong,however,thelevelofcorrelationisweak(R2=0.29)sincelotacceptancerateisalsoinfluencedbyotherfactorsbesidesqualityculture.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 50
5
Figure 27: Lot Acceptance Rate and Deviations RecurrenceSample of 25 plants, all technologies, average annual values
97
98
99
100
0 5 10 15 20 25 30 35
Recurring deviations % of closed deviations
R2 = 0.22 R = 0.47
P-value = 0.017
Lot acceptance rate % of lots dispositioned
R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationshipbetweenXandY)to1(perfectlinearrelationship).Pearsoncoefficient(R)isameasuretowhatextenttwovariablesmoveinthesamedirection.Itvariesfrom0(randomrelationship)to1(perfectlinear relationship) or -1 (perfect negative linear relationship). P-value is probability that correlation is zero (in this case this means there is no linear correlation betweenXandYvariables),valuebelow0.05indicatessignificantresults.
Ahigherdeviationsrecurrencerateiscorrelatedtolowerlotacceptancerateatasignificancelevelof0.017%,withhigherqualityculturevaluesalsoassociatedwithlowerrecurringdeviationsrate.Thelevelofcorrelation(R2=0.22),however,isweak,sincelotacceptancerateisinfluencedbyfactorsotherthandeviationsrecurrence.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 51
5
Figure 28: Action Limit Excursions (Sterile Products) and Quality Culture ValuesSample of 6 plants, steriles, average annual values
0
0.05
0.10
0.15
0.20
0.84 0.88 0.92 0.96
Action limit excursions Per sterile lot dispositioned
Quality culture capabilities Top 2 boxes
R2 = 0.69 R = 83
P-value = 0.04
R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationshipbetweenXandY)to1(perfectlinearrelationship).Pearsoncoefficient(R)isameasuretowhatextenttwovariablesmoveinthesamedirection.Itvariesfrom0(randomrelationship)to1(perfectlinear relationship) or -1 (perfect negative linear relationship). P-value is probability that correlation is zero (in this case this means there is no linear correlation betweenXandYvariables),valuebelow0.05indicatessignificantresults.
Strongerqualityculturevalues(capabilitydimension)alsolinktofeweractionlimitexcursionsforsterileproductmanufactureatasignificancelevelof0.04%.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 52
5
Figure 29: APQR on Time and CAPA Effectiveness RateSample of 24 plants, all technologies, average annual values
60
80
100
40 60 80 100
CAPAs effective % of evaluated CAPAs
APQR on time % all APQRs
R2 = 0.44 R = 0.66
p-value < 0.001
R2measureshowwellvariabilityofgivenmetricXexplainsvariabilityofmetricY.Itrangesfrom0(norelationshipbetweenXandY)to1(perfectlinearrelationship).Pearsoncoefficient(R)isameasuretowhatextenttwovariablesmoveinthesamedirection.Itvariesfrom0(randomrelationship)to1(perfectlinear relationship) or -1 (perfect negative linear relationship). P-value is probability that correlation is zero (in this case this means there is no linear correlation betweenXandYvariables),valuebelow0.05indicatessignificantresults.
CAPAeffectivenessrateisstronglyrelated(significance<0.001%)toAPQRsontime.This relationship may possibly be driven by similar culture and capabilities. This has notbeendemonstrated,however.
Inconclusion,thefollowingmetricsandsurveyvalueshavestatisticallysignificantrelationshipseitherdirectlytothemainexternalqualityoutcome(USrecalls)orviaaninternalqualityoutcomeasshowninFigure22:
f Critical complaints f Lot acceptance rate f Deviations rate f Recurringdeviationsrate f Quality culture values
Reworkrateisstronglylinkedtolotacceptanceratebutisnotincludedintheabovelist.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 53
5
5.9 Relationships at Lower Levels of Significance
Inadditiontotheanalysisoutlinedabove,relationshipsataweakersignificancelevel(morethan5%chancethattherelationshipiscoincidental)werealsoseenbetween:
f Critical complaints and lot acceptance rate (Figure 30) f Critical complaints and US recalls (Figure 31) f Deviationsrecurrencerateandqualityculturevalues(Figure32) f CAPAeffectivenessrateandqualityculturevalues(Figure33)
Figure 30: Critical Complaints and Lot Acceptance RateCritical complaints in selected intervals of lot acceptanceSampleof22plants,finisheddosage,averageannualvalues
Median level of critical complaints for sites with higher and lower lot acceptance rate
>99.64 <99.64
0.54 0.42
<99.24
1.20
<98.50
2.78
Lot acceptance rate %
ppm
Sample size 6 5 5 6
Lot acceptance rate may have a relationship to critical complaints. Values for lot acceptancerateinFigure30arequartileboundaries.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 54
5
Figure 31: Critical Complaints and US RecallsComplaintsforplantswithandwithoutrecallsFinished dosage, average annual values
Critical complaints ppm
0.88
0.54
recalls no recalls
+63%
Sample size 5
26
35
no recalls recalls
-26%
Total complaints ppm
25 5 24 Sample size
Figure31showsthatcriticalcomplaintstrendisconsistentwithnumberofUSrecallsfromtheleft-handgraph,whiletotalcomplaintsgoesintheoppositedirection.Apossiblereasonforthisdifferenceisthatcriticalcomplaints,althoughdifficulttodefineprecisely,couldbemuchlessvariablethantotalcomplaints,whichincludesmany categories, such as subjective defects.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 55
5
Figure 32: Deviations Recurrence Rate and Quality Culture ValuesRecurringdeviationsinselectedintervalsofQualitycultureoverallSample of 9 plants, solids, average annual values
Median level of recurring deviations rate % closed deviations
8.3
20.0
>0.8 <0.8 Quality culture overall Top 2 boxes
Sample size 4 5
Figure32indicatesthatthereissomeevidencethatqualityculturevaluesarerelatedtodeviationsrecurrencerate.Whenthesampleissplitintotwoparts,thosesiteshavingaqualityculturevaluesgreaterthan0.8arefoundtoalsohavealowerrecurringdeviations rate (8.3 versus 20.0)
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 56
5
Figure 33: CAPA Effectiveness Rate and Quality Culture ValuesCAPAeffectivenessinselectedintervalsofqualitycultureSampleof74averagequarterlyvalues,alltechnologies
Median level of CAPA effectiveness for sites with higher and lower culture score
ppm
0.89
>0.8
0.95
<0.8
Sample size 19 18
Significance: 0.06
Quality culture overall Top 2 boxes
+7%
UsingthesamequalityculturevalueasgiveninFigure32of0.8,theremayalsobeadifference(7%observed)inCAPAeffectivenessrate.Thesignificancelevelforthisrelationshipiscloseto,butabove,0.05%(0.06%)andthisisshowninFigure33.
Note:Thequalityculturevalueof0.8alsosplitsthesampleintotwoalmostequalparts.
5.10 Comparisons Where Metrics Are Not Differentiated or Are Inconclusive
Somemetricswerenotdifferentiatedorhadnoconclusiveresultsandtheseincluded:
f APQRontime,whichisalsonothighlydifferentiating,reportedas100%bythemajority of sites.
f Forstabilityfailures,confirmedOOS(productreleaseandincomingmaterials)andUnconfirmedOOS,thepilotsampledidnotyieldclearconclusions.
f Thetechnology-specificmetricswerenotsufficientlytestedduetosmallersamplesize,andsomeofthem(mediafills,environmentalrejects)werenotdifferentiatedbetweensites.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 57
5
5.11 Discussion of Relationships
Combinedfindingsonmetrics,qualityculturesurveyvaluesintermsofrelationships,timespenttocollectmetrics,difficultyofcollectionanddifficultyofdefinitionaresummarizedbelowinTable4.
Table 4: Overview of Findings from PilotSite level data
Value Time Difficulty1Definitions discussion2
US recalls Relationship Low 1
Total complaints rate Inconclusive yet High 2
Critical complaints rate Relationship High 2 Yes
Lot acceptance rate Relationship Moderate 2
Reworkrate Inconclusive yet Moderate 2 Yes
ConfirmedOOSrate Inconclusive yet High 2
Stability failure rate Inconclusive yet Moderate 2
Deviations rate Relationship Moderate 2
Invalidated OOS rate Inconclusive yet Moderate 2
APQRontime Inconclusive yet Low 1
Recurringdeviationsrate Relationship Moderate 3 Yes
CAPAeffectivenessrate Inconclusive yet Moderate 2 Yes
Successfulmediafills Notdifferentiating Low 2
Action limits excursions Inconclusive yet Moderate 3
Environmental rejects Notdifferentiating Moderate 3
Culture Relationship
1 1=veryeasyto4=verydifficult;-siteratingsrelatedtowhetherthedataareavailableintherequestedform,orrequiresrecalculation/aggregation,orcollectionfromfragmentedsources;
2 Metricswheredefinitionsvarysignificantlyacrosscompanies,andfeedbackwasprovidedthroughpilot
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 58
5
FurtherdiscussionofeachcolumninTable4isgivenbelow.
5.11.1 Column 1: Value
AsillustratedinFigure22thefollowingmetrics/valueshavestatisticallysignificantrelationships (as discussed in Section 5.8)directlyorindirectlywithanexternalqualityoutcome(USrecallsorcriticalcomplaints):
f US recalls (outcome) f Critical complaints (outcome) f Lot acceptance rate f Deviations rate f Recurringdeviationsrate f Quality culture
Incolumn1,theabovemetrics/valuesareshownwithgreyshading.
Critical complaints is related to US recalls and may be a relevant measure of external quality,however,therewasmuchfeedbackfromsiteleadersfromparticipantcompaniesthatthedefinitionofcriticalcomplaintsledtodifficultyininterpretationofvaluestosubmitandasnotedearliertherewasaonlyaverysmallnumberofrecallsfromthepilot study.
Lotacceptancerate,deviationsrate,deviationsrecurrencerateandqualityculturevaluesallhaverelationships(somestatisticallysignificant)toqualityoutcomes(critical complaints and US recalls).
As stated at the beginning of this section, it cannot be stressed too strongly that the statistically significant relationships observed in the Wave 1 Pilot sample do not indicate cause.Furthertestingandstudiesarerequiredtobetterunderstandtheserelationships.
SomemetricshadinconclusivecomparisonswithothermetricsandfurthertestingisrequiredtoindicatethepresenceorabsenceofrelationshipsasshowninTable4.
Somemetriccomparisonsfortechnology-specificmetrics(successfulmediafillsandenvironmentalrejects)werenotdifferentiated;thesearehighlightedwithorangeshading in column 1.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 59
5
5.11.2 Column 2: Time
Timespentcollectingeachindividualmetricistabulatedincolumn2.Metricswithlowestamountoftimeareshadedgreyandthosewiththehighesttimeareshadedorange.Somepossiblereasonsfordifferencesare:
f Complaintsareusuallyreceivedcentrally,andeffortisrequiredtounderstandthe complaint and distribute to a particular site in a supply chain, and then for that site to categorize the complaint and submit a value. Also some complaints mayneedtobeforwardedtoanumberofdifferentsites(e.g.,toapackagingsite if the complaint is related to the packaging operations).
f AconfirmedOOSmetricmaybehardtocollectonasitebasissinceitisusuallycollectedonaproductbasisandeffortisrequiredtoaggregatetoasitelevel.
f APQRsontimemetricsareprobablyeitheralreadycollectedoreasytocollect.
f USrecallsarealownumberandofsuchhighimportancethatthemetriciseasyto collect.
f Forsterileproductmanufacturingsites,thesuccessfulmediafillsmetricisarelativelylownumberandeasytocollect.
EstimatesoftimetocollectQualityCultureSurveydatawerenotcollectedintheoveralleffortestimates,sinceeventheapproximately5minutesrequiredforeachindividualresponsewouldstillpresentalargeeffortgiventheoverallnumber(10,300)of survey respondents.
5.11.3 Column 3: Difficulty
Theserankingsaretheparticipatingsites’estimatesofthedifficultyofcollectingametricasshowninFigure16.
Mostdifficultwastherecurringdeviationsrate,sincethismetricdoesnotappeartobecollectedroutinelybymostsites.AsshowninFigure17,thereisnotnecessarilyacorrelationbetweendegreeofdifficultyofcollectingametricandmedian time for collection.
5.11.4 Column 4: Definitions
Inthelastcolumn,metricsaregivenwheretheMcKinseysupportteamspentmosttimeexplainingametricdefinitionandwhere,fromfeedback,definitionsvariedmostbetweencompaniesandbetweenacompanydefinitionandthedefinitionusedinthepilot.Themetricsmostdifficulttodefinewere:
f Critical complaints rate f Reworkrate f Recurringdeviationsrate f CAPAeffectivenessrate
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 60
5
5.12 Complaints Analysis
Thedatabasewasexaminedtoevaluaterelationshipsbetweencriticalcomplaints,lotacceptancerateandtypeofproductandtechnology.Datawereplottedinascatter plot as given in Figure 34.
Figure 34: Scatter Plot of Critical Complaints, Lot Acceptance Rate, Technology and Type of ProductProductleveldata,retrospectiveperiod(12m.),finisheddosageplants
0
200
400
600
800
1,000
100 99 98 97 96 95 94 93 92 91 90
Critical complaints rate1 ppm
Lot acceptance rate 2
% lots dispositioned
SOURCE: ISPE Quality Metrics pilot, January 2015
Size of bubble indicate volume of product in packs
High rejects but few critical complaints. This “conservative” segment represents: 16% of overall volume 41% of Gx volume 56% of steriles volume
High critical complaints, high rejects. This “issues” segment represents: 36% of overall volume 53% of OTC volume 63% of liquids and creams volume
High critical complaints, low rejects. This “misses” segment represents: 15% of overall volume (with roughly proportionate share within each technology or type of product)
Few critical complaints, low rejects. This “healthy” segment represents: 33% of overall volume 48% of Rx volume 63% of solids volume
1 Productswithover1000criticalcomplaintspermilionpackshavebeenomittedonthechart2 Productswithlessthan90%lotacceptancehavebeenomittedonthechart
The red lines represent the 80th percentile boundaries for lot acceptance rate (99.24,lowertotheleft)andcriticalcomplaints(0.12,lowerbelowline).
Thisshowsthatproductsvaryintheirrejectsandcomplaintsprofiledependingonboththetechnologyandthetypeofproduct.Itisthereforehardtodrawanyspecificconclusions at this time from this analysis and available data set.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 61
5
5.13 Analysis of Product-Based Metrics
Product-basedmetricswereanalyzedforthecomplaintsandlotacceptanceratemetrics.Theanalysisindicatedthat“tail”products(90%ofproductsrepresenting30%ofvolume)representadifferentlevelofcriticalcomplaintsandrejectsdependingontechnologyasshowninFigure35andFigure36.
Figure 35: Critical Complaints and Volumes for Product-Based DataProduct level data1
7058
3042
Critical complaints
Volume (packs)
72
28
28
72
Volume (packs)
Critical complaints
70
47
30
53
100%
Volume (packs)
Critical complaints
Solids Steriles Other
High-volume products XX # products in group
100
7 25
157 458
63
1 Excludesinactiveproducts(withnopacksreleasedinreportingperiod)
Themultipletailproducts,representing30%ofvolume,resultin42%ofcriticalcomplaintsforsolids,70%ofcriticalcomplaintsforsterileproductsand53%of critical complaints for other technologies.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 62
5
Figure 36: Rejects (from Lot Acceptance Rate) and Volumes for Product-Based DataProduct level data
High-volume products xx # products in group
Solids API and biologics OtherSteriles
30
70
34
66
30
70
75
25
29
71
69
31
30
70
37
63
Volume(lots disp.)
Lotsrejected
Volume(lots disp.)
Lotsrejected
Volume(lots disp.)
Lotsrejected
Volume(lots disp.)
Lotsrejected
558
102
138
92
18
98
14
52
100%
Forrejectsobservedaspartoflotacceptancerate,30%ofthevolumegave34%rejectsforsolids,70–75%ofrejectsforbothsterileproductanddrugsubstancesitesand37%rejectsforothertechnologies.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 63
5
5.14 McKinsey Analytical Effort and Observations
AnalysisbyMcKinseyregardingtheamountofeffortthattheirsupportpersonnelhadperformedwasestimatedas:
f Eachsiterequiredonaverage22hoursofdedicatedsupport—explainingthedatarequirements,answeringquestions,reviewingandverifyingthesubmitteddatathroughdiscussionswiththesite.
f The more structured and supported the data submission process is, the more accurate the received data.
f Checkingsubmitteddataforcoherence/accuracywasrequiredevenafterseveral submission cycles.
f Preparation of submission templates, databases and process, and analyzing thegathereddatarequiredapproximately400additionalhours
TheabovemayindicatethelevelofsupportthatFDAwillberequestedtoprovide,at least for early rounds of data collection.
DuringthecourseofthePilotthereweremanyinteractionsbetweenparticipatingcompaniesandMcKinseypersonnelandthefollowingsuccessfactorswereobserved:
f Good definitions: To make data submission as easy as possible and minimize theneedforfollow-updiscussion,metricsmustbedefinedtoaveryhighstandard.Industry-consensusdefinitionsalmostalwaysdifferfromthosebeingusedbycompanies;henceitisnecessarytoallowcompaniestoadjust.
f Strong support:Experienced,dedicatedsupportwasrequiredthroughoutthecollectionperiodtoanswerquestionsquicklyandmakenecessaryclarifications.
f Engaged and committed participants: Given that participating in the pilot wasvoluntary,itwasclearthatparticipantcompanystaffwerecommittedandknowledgeable;mostcompaniesalsohadmaturesystems.UsingMcKinseypersonnel and teleconferences of participant company site leaders permitted muchsharingandlearningbetweencompanies.
f Built-in checks: Submitteddatarequiredcarefulcheckingtoensureconsistency and accuracy.
f Feedback from participating companies:McKinseycollectednonconfidentialandcross-companycommentsaboutpilotdesignandoperation.DefinitiondifferencesbetweenISPEandparticipatingcompaniesfor“recurringdeviationsrate” generated considerable feedback, for example.
Thistypeoffeedbackreinforcestheneedtohaveprecise,agreeddefinitions.Thereisthepossibilitythatevenwithprecisedefinitions,differencesinproductandprocessflow,couldleadtoundesirablevariationincompanies’submissions.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 64
6 Output and Lessons Learned from ISPE Quality Metric Wave 1 Pilot
ThissectiondiscussessuccessfactorsfortheWave1Pilotandanoverviewoffindingssuchasasummaryofeffortrequiredandcomparisonofsite-andproduct-based data.
TheISPEQualityMetricsProjectTeamhaveproducedthefollowinglessonslearnedandoutputsfromtheWave1PilotbasedonfindingsanddatageneratedduringthePilot,commentsmadebyparticipantsaswellascommentsbyMcKinseypersonnelcomparingtheirexperienceswiththisISPEPilotwithotherbenchmarkingexercises.
6.1 Success Factors
Thefollowingarelistedassuccessfactorsforthepilot:
f Metrics lists prealigned during industry meetings. f Precisedefinitions,developedwithinputfrommultiplecompanies. f Engagedandcommittedparticipantswithmaturesystems. f Acceptabletosubmit“goodenough”data. f Strongcollaborationacrosscompaniesandbetweenexpertstosupportaprocessthatallowslearningandcontinuouslyimprovingdataaccuracy.
f Strong McKinsey Support, providing: – Structureddatasubmissionwithdetailedguidanceonhowtoreportthedata. – Ability to comment on data points to enable interpretation. – Experienced,dedicatedsupportforquestionsandclarificationsduringandthroughoutthedatacollectioneffort.
– Built-inchecksandjointreviewbetweenMcKinseyandtheparticipatingcompany to ensure consistency and accuracy of data.
– Transparencythroughouttheprocess,engagingparticipantsinfrequentdebriefs and discussions.
Thesefactorssupportstartingwithaqualitymetricsprogramcontainingarelativelysmallnumberofwell-definedmetricswithalearningperiodtorefinescope,definitionsandprocess.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 65
6
6.2 Definitions
Definitionsareextremelyimportant:
f Definitionsmustbeexact:Denominatorsinparticulararehighlysensitivetoissuesaroundlotaggregationandfinaldisposition.
f Evencommontermslike“lot,”“deviations,”“complaints”and“reviews”mustbespecifiedingreatdetailtominimizemultipleinterpretations.
f Evenwithdetaileddefinitions,supportandansweringquestionsthroughoutthe process is necessary to ensure more accurate data submission.
f Standardizeddefinitionswilldifferfromcurrentcompanydefinitions,thusrequiringadditionalwork.
f Productandprocessdifferenceswillgeneratedifferencesandvariationsin metric ranges.
f Commentary on data points is essential to interpretation and analysis. f Somevariationmustbeexpectedduetodifferencesinproduct,processflows
and product/process complexity.
Thepilotshowedthatstandardizingmetricsdefinitionsacrosscompaniesisfeasible.
6.3 Metrics Collection by Site and by Product
Attemptsweremadetogenerateproduct-leveldata,however,therearechallenges:
f Thepilotcollectedmetricsonlywithinagivensite,notacrossmultiplesitesor across a full supply chain.
f Acompany’sabilitytocollectdatabysiteand/orbyproductdiffersbymetricandhowtheirsystemsaresetup.
f Most companies collect data at site level and may have to manually disaggregate data in order to report at a product level.
f Some metrics data collected at sites cannot be easily allocated to a product (e.g., deviation rate).
f Some companies report some metrics (e.g., OOS) at a product level and have to aggregate data to report at a site level.
f SomedatacollectedatproductlevelmaybederivedfromAPRs. f Dependingonthesizeandcomplexityofthesite,APRpreparationmaybespreadbyproductondifferentcycletimesduringtheyear.
f Complaints data are generally gathered centrally (corporate level) and distributed to sites for investigation.
Insummary,fewcompaniesaggregatemetricsacrossthesupplychaintobeableto report at product/application level.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 66
6
6.4 Industry Effort
Industryeffortissummarizedas:
f On average the participating sites in the Wave 1 Pilot spent ~90 hours in pure datacollectiontime.For12,000siteswithFEInumber(6,000API/FinishedDosageand6,000Labs/DrugSubstance/‘other’)attypicalqualitylaborcost,collectingthisamountofdatawouldcost the industry an additional ~$35 million annually. – This conservative estimate does not include several factors that could
bring the cost of such a program to $100+ million, such as: f The 90 hours observed might be underestimated for sites that did not report
all products at that site. f Workloadestimatesfor“goodenough”datasubmissionvs.anofficialsubmission. f Timeforinternaldiscussions,managementreviewandabove-siteguidance
not included. f Theneedfornew/modifiedITsystemswasnotincludedinWave1. f Participantshadflexibilitytoprovidethemostpragmaticdataset
(e.g., for all products at site or only those for the US market). f Datawasprovidedwithineachsiteandnot through the full product supply chain. f Most participants had mature systems and capabilities regarding
performance measurement. f Majorityofsiteswerefromdevelopedcountries.
Theprogramscopeanddesigncaninfluencetheseindustrycostssignificantly.
6.5 McKinsey Analytical Effort
TherewassubstantialeffortfromMcKinsey:
f Eachsiterequiredonaverage22hoursofdedicatedsupport—explainingthedatarequirements,answeringquestions,reviewingandverifyingthesubmitteddatathroughdiscussionswiththesite.
f The more structured and supported the data submission process is, the more accurate the received data.
f Checkingincomingdataforcoherence/accuracywasrequiredevenafterseveraldata submission cycles.
f Preparation of submission templates, databases and process, and the analysis ofthegathereddatarequiredapproximately400additionalanalyticalhours.
As already mentioned in Section 5.14, the above points may indicate the level of support thatFDAwillberequestedtoprovide,atleastforearlyroundsofdatacollection.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 67
7 Proposals
BasedonfindingsfromtheWave1Pilot,intermsofeaseandeffortofcollectionandsubmissionofmetrics,itissuggestedthatfuturequalitymetricsprogramsstartwitharelativelysmallnumberofstandardizedmetrics.
FromtherelationshipfindingsdiscussedinSection 5a“startingset”offivemetricsareproposedforinclusioninfutureindustryqualitymetricsprograms:
1. Lot acceptance rate (normalized by lots dispositioned) at site level.2. Lotacceptancerate(normalizedbylotsdispositioned)atproductlevelwithin
a site.3. Critical complaints (normalized by number of packs released) at product level
byapplication,notbrokendownbysite.4. Criticalcomplaints(normalizedbypacksreleased)atsitelevel,undifferentiated
by product.5. Deviations rate at site level.
RationaleforincludingthesemetricsinaWave2Pilotaregivenbelow.
7.1 Rationale for Metrics Proposed as a Starting Set for Wave 2 Pilot
f InmostcasesthemetricwasalreadycapturedintheWave1Pilot,andcontinuedmonitoring is desired over a longer timeframe and broader set of companies, technologies, regions.
f Themetricselecteddemonstratedastatisticallysignificantrelationshiptooneofthefollowing: – Deviations recurrence – Quality culture values – Critical complaints – Lot acceptance rate
f It has been demonstrated to be relatively easy to collect and submit. f Itwasdeemedanimportantmetricfordeterminingsitequalityperformance. f Itwillassistthecompanytoidentifycontinualimprovementopportunities. f While the critical complaints metric (normalized by number of packs released atproductlevelbyapplication,notbrokendownbysite)wasnot included in the Wave 1 Pilot, it is thought to have merit and should be explored by product application.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 68
7
ThestartingsetofmetricsarenowbeingconsideredforfurtheranalysisinanISPEQualityMetricsWave2Pilotthatwillincludeanextendedtimeperiodofprospectivedata collection and increased sample size of participating companies. The momentum andknowledgegainedfromtheWave1PilotwillbeleveragedtofacilitateanearlytransitiontoWave2.AttheQualityMetricsSummitinBaltimore,April2015,notificationwasgivenforbothnewandcontinuedparticipationinaWave2PilotwithatargetenrollmentcommencementdateofJune2015.FurtherdesignoftheWave2Pilotwillthengetunderway.
Inadditiontothestartingsetoffivemetricslistedabove,othermetricsofinterestunder consideration for a Wave 2 Pilot include:
f Deviations recurrence rate: Howtoachieveaconsistentandacceptedindustrydefinitionandpractice.
f Unconfirmed OOS: Test on a bigger sample its relation to culture, particularly useful for laboratories, and assess against outcomes.
f Quality culture: Howbesttoassesstheinfluenceofqualitycultureonqualityperformance outcomes at industry scale.
Briefingsessionswithexistingparticipatingcompaniesareunderwaytodiscusstheimplications for ongoing participation. Many other companies in attendance at the QualityMetricsSummitwhoexpressedawillingnesstogetinvolvedinfuturepilotstudieswillbecontactedintheneartermregardingpotentialenrollment.
TheWave2Pilotwillgiveparticipatingcompaniesadeeperunderstandingofmetricsdefinitions,theopportunitytocontributetothefinalpilotstudydesign,andthechanceto experience the challenges of a centralized metrics submissions process. In addition, participantswillhaveaccesstoanindustrybenchmarkingreportthatwillallowthemtoexaminetheirprogresswithrespecttotheirpeers.Finally,participationalsoprovidesopportunities to enhance the maturity of their internal metrics programs.
Those interested in receiving further details should contact the ISPE Quality Metrics Initiative at [email protected].
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 69
8 Conclusions
TheobjectivesoftheWave1Pilotwereachievedandconsequentlythepilotisconsidered a success.
Evidenceofseveralstatisticallysignificantandinterestingrelationshipswasfoundbetweencompanyqualityperformance,prevailingqualitycultureandkeypatient-relatedqualityoutcomes.TheinsightsandknowledgegainedfromthisWave1Pilotconfirmthebenefitsofutilizingqualitymetricstomonitor,createtransparencyanddrive enhancement in pharmaceutical manufacturing.
Industryengagementthroughoutthisprocesshasconfirmedthatmanysitesandcompanies, including those participating in the Wave 1 Pilot, are already using metrics to drive and monitor internal continual improvement programs. This pilot wasanattempttodevelop,collectandanalyzeastandardizedsetofmetricsandhas indicated that this is achievable. While there are inherent challenges and costs associatedwiththisexercise,thepilotwasabletoidentifyseveralkeybenefitsalso.The pilot therefore accomplished its primary objectives and has set the stage for additionalworkinWave2.
TheserecommendationswerediscussedwithindustryrepresentativesandFDAattheISPEQualityMetricsSummitandbroadsupportwasgiventotheoutcomesandfutureplans.Asignificantopportunityfortrustbuildingbetweentheindustryand the agency has also been realized though this Wave 1 Pilot.
CommunicationswillcontinuewithallpartiesastheISPEQualityMetricsInitiativefurther develops its plans for the Wave 2 Pilot.
Ten years ago Dr. Janet Woodcock outlined the ultimate goal of the desired state [18] for pharmaceutical manufacturing. This Quality Metrics Initiative Pilot has provided earlysupportofthisjourneytowardpharmaceuticalexcellence.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 70
9 References
1. USDepartmentofHealthandHumanServices:FoodandDrugAdministration,etal.“GuidanceforIndustry:QualitySystemsApproachtoPharmaceuticalCGMPRegulations.”September2006. http://www.fda.gov/downloads/Drugs/.../Guidances/UCM070337.pdf
2. .“PharmaceuticalGMPsforthe21stCentury—ARisk-BasedApproach.”FinalReport.September2004. http://www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/Manufacturing/ QuestionsandAnswersonCurrentGoodManufacturingPracticescGMPforDrugs/UCM176374.pdf
3. .“GuidanceforIndustry:PAT—AFrameworkforInnovativePharmaceuticalDevelopment, Manufacturing, and Quality Assurance.” September 2004. http://www.fda.gov/downloads/Drugs/Guidances/ucm070305.pdf
4. .“GuidanceforIndustry:ProcessValidation:GeneralPrinciplesandPractices.”Revision1.January2011. http://www.fda.gov/downloads/Drugs/Guidances/UCM070336.pdf
5. InternationalConferenceonHarmonisationofTechnicalRequirementsforRegistrationofPharmaceuticalsforHumanUse.ICHHarmonizedTripartiteGuideline.“PharmaceuticalDevelopment:Q8(R2).”Step4version,August2009. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q8_R1/Step4/Q8_R2_Guideline.pdf
6. .QualityRiskManagement:Q9.”Step4version,9November2005. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q9/Step4/Q9_Guideline.pdf
7. .“PharmaceuticalQualitySystem:Q10.”Step4version,4June2008. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q10/Step4/Q10_Guideline.pdf
8. .“DevelopmentandManufactureofDrugSubstances(ChemicalEntitiesand Biotechnological/Biological Entities): Q11.” Step 4 version, 1 May 2012. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q11/Q11_Step_4.pdf
9. GovernmentPrintingOffice.FoodandDrugAdministrationSafetyandInnovation Act (FDASIA). Publication L. 112–144. July 9, 2012. http://www.gpo.gov/fdsys/pkg/PLAW-112publ144/pdf/PLAW-112publ144.pdf
10. BrookingsInstitution.“MeasuringPharmaceuticalQualitythroughManufacturingMetricsandRisked-BasedAssessment,”ExpertWorkshop,1–2May2014 http://www.brookings.edu/events/2014/05/01-measuring-pharmaceutical-quality
11. McKinsey&Company.“PharmaOperationsBenchmarkingofSolids(POBOS).” http://solutions.mckinsey.com/pobos/
12. ISPE.“ISPEProposalsforFDAQualityMetricsProgram—Whitepaper.”20 December 2013. www.ISPE.org
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 71
9
13. USDepartmentofHealthandHumanServices:FoodandDrugAdministration.“GuidanceforIndustry:Self-IdentificationofGenericDrugFacilities,Sites,and Organizations.” August 2012 http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm316721.htm
14. ISPE.“ISPEDrugShortagesPreventionPlan:AHolisticViewfromRootCauseto Prevention.” October 2014. www.ISPE.org/DrugShortagesPreventionPlan
15. USDepartmentofHealthandHumanServices:FoodandDrugAdministration.“FoodandDrugAdministrationDrugShortagesTaskForceandStrategicPlan;RequestforComments.”Federal Register, Docket No. FDA-2013-N-0124. 12 February 2013. http://www.gpo.gov/fdsys/pkg/FR-2013-02-12/html/2013-03198.htm
16. Woodcock,Janet.“TheQualityRevolution:TheFutureofPharmaceuticalManufacturingandItsRegulation.“PresentedattheISPEQualityMetricsSummit, Baltimore, MD, 22 April 2015.
17. http://rx-360.org/Portals/1/Guidance/FDAAnnualInspectionReportonEstablishments/2015AnnualReportonInspofEstablishments inFY2014Final.pdf
18. Woodcock,J.“TheConceptofPharmaceuticalQuality.”American Pharmaceutical Review47(6):1–3,2004.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 72
Appendix 1Appendix 1Definitions of Quantitative Metrics Used in Pilot
Lot acceptance rate
Metric Definition
Lot acceptance rate = Total lots released for shipping out ofthetotalfinallydispositionedlotsfor commercial use in the period
f Total lots dispositioned = total number of lots for commercial use produced and/orpackagedonsitethatwentthroughfinaldispositionduringtheperiod,i.e.werereleasedforshippingorrejected(fordestruction).Rejectionsshouldbecountedasfinaldispositionregardlessatwhatproductionstagetherejectionoccurred.Releaseisonlyfinalreleaseforshipping.Excludeslotsthathavebeensentforreworkorputonhold/quarantinedinthisperiodandhencearenotfinallydispositioned.Excludeslotsthatarenotproducedor packaged on site, but just released for CMOs.
f Totallotsrejected=totalfulllotswererejectedforqualityreasons.Rejectedmeansintendedfordestructionorexperimentaluse,notforreworkorcommercialuse.Rejectionsshouldbecountedregardlessatwhatproductionstage the rejection occurred.
f Total lots released (accepted) = total lots dispositioned less total lots rejected.
Complaints Rate (Total and Critical)
Metric Definition
Total complaints rate = Total complaints received in the reporting period, related to the qualityofproductsmanufacturedinthe site, normalized by the number of packs released
f Packsreleased=Totalnumberofpacks(finalproductformthatleavestheplant, one level less than tertiary packs, most usually it is secondary packaging unit e.g. pack of blisters or bottle in carton pack) released in the period.
f Total complaints = All complaints received in the reporting period, related tothequalityofproductsmanufacturedinthesite,regardlesswhethersubsequentlyconfirmedornot.Allcomplaintsreceivedbythesiteshouldbecounted,evenifacomplaintaffectsmorethan1site,orifeventuallytherootcause analysis attributes the issue to another site. Complaints related to lack ofeffectshouldbecountedaswell.
Critical complaints rate = All critical complaints, normalized by the number of packs released
f Criticalcomplaints=Post-distributionproductqualitycomplaintswhichmayindicateapotentialfailuretomeetproductspecifications,mayimpactproduct safety and could lead to regulatory actions, up to and including product recalls. Critical complaints include those that potentially could lead toFDAnotification(e.g.,FieldAlertReports,BiologicalProductDeviationReports).Critical(orexpedited)complaintsareidentifieduponintake,whethersubsequentlyconfirmedornot,basedonthedescriptionprovidedby the complainant, and include, but may not be limited to: – i. Information concerning any incident that causes the drug product
or its labelling to be mistaken for, or applied to, another article. – ii.Informationconcerninganybacteriologicalcontamination,oranysignificant
chemical, physical, or other change or deterioration in the distributed drug product, or any failure of one or more distributed batches of the drug producttomeetthespecificationestablishedforitintheapplication.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 73
Appendix 1
OOS Rate, Stability Failure, Invalidated (Unconfirmed) OOS Rate
Metric Definition
ConfirmedOOSrate= TotalconfirmedOOS(testresultsthatfalloutsidethespecificationsor acceptance criteria), out of all lots dispositioned by the lab during the period
f Total lots tested/dispositioned by the lab = total number of lots used for commercial production that are tested and dispositioned out of the lab in the period, i.e., have a QC pass or fail decision on them. Includes: – Lots for release testing (counted as 1 lot, even if sampled separately for
chemical and microbiological testing, or for in-process analytical testing inlaboronshopfloor).
– Lots of incoming materials for analytical testing (count 1 per each analyticallytestedrawmaterialand/orpackagingmateriallot).Includeswaterusedasrawmaterial.
– Lots for stability testing in that period (counted as 1 per each timepoint and condition sampled per the approved stability protocol).
– Does not include environmental monitoring samples. f ConfirmedOOS=alltestresultsthatfalloutsidethespecificationsoracceptancecriteriaestablishedindrugapplications,drugmasterfiles(DMFs),officialcompendia,formularyorappliedbythemanufacturerwhenthereisnotan‘official’monograph.
Stability failure rate = TotalconfirmedOOSrelatedto stability testing
f SubsetoftheConfirmedOOSrate–basedonstabilitylotstestedandconfirmedOOSrelatedtostabilityonly.
Invalidated(unconfirmed)OOSrate=TotalunconfirmedOOS,outofalllots tested during the period
f UnconfirmedOOS=allOOSminusconfirmedOOS(seethedefinitionofconfirmedOOS).
US recall events (Total and by Class)
Metric Definition
Recallrate f Recallevents=allUSmarketrecallevents. f By class = all US market recall events, class I and II. f Recalledlots=Includelotsrecalledeithervoluntarilyorbyregulatoryorder(recallimpliesphysicalremovalofproductfromfield,notjustafieldactionor correction). Include US market recalls only.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 74
Appendix 1
Right First Time (Rework/Reprocessing)
Metric Definition
RFT(rework/reprocessingrate)=Total lots released that have not beenreworkedorreprocessedoutofthetotalfinallyreleasedlotsforcommercial use in the period
f Total lots released (accepted) = total lots dispositioned less total lots rejected (seethedefinitionofLotacceptancerate).
f Totallotsreworkedorreprocessed=alllotsthathavegonethroughrework(using alternative process) or reprocessing (using again the original process) beforethatfinaldispositioninordertomeetrequirementsforrelease.Onlycountreworkorreprocessingnecessitatedbyqualityissues(forexamplecontractmanufacturingsitesshouldexcludereworkduetocustomerorderchanges).Ifalotwassentforreworkandreceivedanewlotnumber,itshouldstillbecountedasundergonereworkwhenfinallydispositioned.
APQRs on Time
Metric Definition
APQRcompletedontime=Number of Annual Product Quality Reviewsintheperiodthatwerecompleted by the original due date, normalized by all products subject toAPQR
f ProductssubjecttoAPQR=TotalnumberofproductssubjecttoAnnualProductQualityReviews-annualevaluationsofthequalitystandardsofeach drug product to verify the consistency of the process and to highlight any trends in order to determine the need for changes in drug product specificationsormanufacturingorcontrolprocedures(asrequiredbyCFRSec.211.180,Generalrequirements,section(e)andICHQ7,GMPsforAPIs,section 2.5 or EU Guidelines for Good Manufacturing Practice for Medicinal ProductsforHumanandVeterinaryUse,Chapter1,PharmaceuticalQualitySystem, section 1.10). Does not include the data packages that a site preparestoitscustomerswhenactingasaCMO.
f NumberofAnnualProductQualityReviewsontime=completedbytheoriginal due date.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 75
Appendix 1
Recurring Deviations Rate
Metric Definition
Recurringdeviationsrate= Number of deviations that have re-occurred during the preceding 12 month period out of all closed deviations
f Number of deviations = Any major or minor unplanned occurrence, problem, or undesirable incident or event representing a departure from approved processes or procedures, also includes OOS in manufacturing or laboratory or both. Please count only deviations that have been closed/resolved in the period.Deviationsfromoneperiod,forwhichtheinvestigationwasclosedin the next period, should be counted in the latter period.
f Recurringdeviations=Numberofdeviationsforwhichduringthe12monthperiod preceding each deviation, at least one other deviation has occurred withthesamerootcausewithinthesameprocessand/orworkarea.Ifredundant/duplicativeprocessesorequipmentexist,pleaseconsiderdeviationeventscommontothegrouping/workcenterasrecurring(stillwithinthe12 month timeframe). For example, if a deviation for missing desiccant occurstwice,ontwoseparatepackaginglineswithcomparableequipment/systems, it should be counted as recurring (i.e. as 2 same deviations, rather than1differentforeachline).
CAPA Effectiveness Rate
Metric Definition
CAPAeffectivenessrate= NumberofCAPAseffectiveoutofallCAPAswitheffectivenesscheckin the reporting period
f CAPAswitheffectivenesscheck=NumberofCAPAsevaluatedforeffectivenessinthereportingperiod.AllCAPAsshouldbecounted,includingthose related to inspection or audit observations.
f CAPAseffective=thoseevaluatedCAPAswherethequalityissuesubjectoftheCAPAwasresolved,and/orhasnotreoccurred,andtherehavebeenno unintended outcomes from the CAPA implementation.
Media Fill Failures (sterile/aseptic only)
Metric Definition
Mediafillrate= Numberofmediafillsdispositionedassuccessfuloutofallmediafillsto support commercial products dispositioned during the period
f Mediafills=Totalnumberofmediafills(regardlessofnumberofrunsineach)tosupportcommercialproductsthatweredispositioned(assuccessfulorfailed)duringtheperiod.Ifthemediafillwasdispositionedasfailureandarerunwasneeded,thatrepeatiscountedasaseparatemediafill.Includesallmediafills-bothforinitialandperiodicqualifications.
f Successfulmediafills=Allmediafillsthatwerenotdispositionedasfailures.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 76
Appendix 1
Environmental Monitoring (sterile/aseptic only)
Metric Definition
Environmental monitoring
f Lotswithactionlimitexcursions, normalized by all sterile dispositioned lots
f Lots rejected due to environmental monitoring reasons, normalized by all sterile dispositioned lots
f Steriledispositionedlotsduringtheperiod(seedefinitionforLotacceptance rate).
f Lotswithlimitexcursions=Allsteriledispositionedlotsduringtheperiodthat had associated investigations related to exceeding environmental monitoring action limits. If a lot had more than 1 such investigation please countonly1perlot.Ifaninvestigationhasaffectedmultiplelots,pleasecount each lot separately. Action limit is an established microbial or airborne particlelevelthat,whenexceeded,shouldtriggerappropriateinvestigationand corrective action based on the investigation.
f Rejectedlotsduetoenvironmentalmonitoringreasons=Allsteriledispositionedlotsduringtheperiodthatwererejectedforexceedingenvironmentalmonitoringactionlimits.Rejectedmeansintendedfordestructionorexperimentaluse,notforreworkorcommercialuse.Rejectionsshouldbecountedregardlessatwhatproductionstagetherejection occurred.
Deviations Rate
Metric Definition
Deviations rate = Number of deviations closed during theperiodoutofthetotalfinallydispositioned lots for commercial use in the period
f Number of deviations = Any major or minor unplanned occurrence, problem, or undesirable incident or event representing a departure from approved processes or procedures, also includes OOS in manufacturing or laboratory or both. Count only deviations that have been closed/ resolved in the period. Deviationsfromoneperiod,forwhichtheinvestigationwasclosedinthenext period, should be counted in the latter period.
f Deviations rate = Number of deviations closed during the period out of the totalfinallydispositionedlotsforcommercialuseintheperiod.
f Total lots dispositioned = total number of lots for commercial use produced and/orpackagedonsitethatwentthroughfinaldispositionduringtheperiod,i.e.werereleasedforshippingorrejected(fordestruction).Rejectionsshouldbecountedasfinaldispositionregardlessatwhatproductionstagetherejectionoccurred.Releaseisonlyfinalreleaseforshipping.Excludeslotsthathavebeensentforreworkorputonhold/quarantinedinthisperiodandhencearenotfinallydispositioned.Excludeslotsthatarenotproducedor packaged on site, but just released for CMOs.
Report from ISPE Quality Metrics Pilot Program / June 1 – April 2015 77
Appendix 2 Survey Questions
Quality Culture Questions
Strongly agree Agree Disagree
Strongly disagree
I can’t answer this question
Cap
abili
ties
Patient focus:Iknowwhichparametersofourproducts are particularly important for patients
Training: The training I have received clearly helps metoensurequalityintheendproduct
Problem Solving:Alllineworkersareregularlyinvolvedin problem solving, troubleshooting and investigations
Gov
erna
nce
Recognition: We recognize and celebrate both individualandgroupachievementsinquality
Metrics:Up-to-datequalitymetrics(e.g.defects,rejects, complaints) are posted and easily visible near each production line
Knowledge:Eachlineworkercanexplainwhatlinequalityinformationistrackedandwhy
Continual Improvement: We are regularly tracking variations in process parameters and using them to improve the processes
Lead
ersh
ip
Coaching:Supervisorsprovideregularandsufficientsupportandcoachingtolineworkerstohelpthemimprovequality
Dialogue:Wehavedailyqualitymetricsreviewsandqualityissuesdiscussionsontheshopfloor
Gemba:Managementisonthefloorseveraltimesa day both for planned meetings and also to observe and contribute to the daily activities
Min
dset
s
Awareness:Everylineworkerisawareofthebiggestqualityissuesontheirlineandwhatisbeingdoneabout them
Responsibility:Allemployeesseequalityandcompliance as their personal responsibility
Inte
grity
Openness:Iamnotafraidtobringqualityissuesto the management’s attention
Ethics:PeopleIworkwithdonotexploittotheiradvantage inconsistencies or ‘grey areas’ in procedures
Motivation: All employees care about doing a good jobandgotheextramiletoensurequality
Report from ISPE Quality Metrics Pilot Program / June 1 – April 2015 78
Appendix 2
Process Capability Questions f Doyoumeasurethattheprocessremainsinastateofcontrol(thevalidatedstate)duringcommercialmanufacturing?(yes/no) f Forwhat%ofproductsaretheyapplied(basedonyourtotalnumberofproductsasreportedindatabysite)-excluding
packaging operations f Ifnotappliedon100%ofproducts,howdoyouchoose/segregate/prioritizeonwhichproductstoapplythesemetrics?(openquestion)
Please indicate which metric or metrics do you use for ongoing monitoring and to what parameters do you apply them CpK Ppk
Tolerance interval Box Plots
Trending of CQAs
Other (specify which in the comments field)
AppliedtoCQA(criticalqualityattributes tested in the lab)
Applied to IPC (in-process control) checks
Applied to CPP (critical process parameters)
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 79
Appendix 3 Examples for Site and Product Data Collection Templates
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 80
Appendix 4Case Study Company A
Introduction
Company A, a participant in the ISPE Quality Metrics Wave 1 Pilot, has a large portfolio of over-the-counter (OTC) products manufactured and distributed globally:
f 8,000+rawmateriallotstestedperyear. f 20,000+ intermediate bulk, packed, and labeled products tested annually. f 750+millionpacksproducedandreleased.
Given the extent of site-level activity, one of Company A’s motivations for participating inaqualitymetricsprogramisitspotentialtoreduceinspectionfrequencies.Unlikeotherpilotparticipants,however,CompanyAwillnotbenefitfromexpeditedapprovalsor improved efficiency in post-approval changes due to the nature of its OTC- and monograph-based business.
CompanyAalsounderstandsthatcollectingqualitymetricsmayhelpreducedrugshortages. Indeed, the ISPE Drug Shortages Prevention Plan [14]states“well-definedmetricstailoredtoproactivelyidentifythepotentialriskofashortage,willhelpmitigateloomingshortages.”Theplanacknowledgesthataprescriptivelyselectedsetofmetricsmaynotalwaysidentifydrugshortages,however.Forthisreason,itrecommends that each metrics program should be tailored to the supply chain situation it addresses.
Finally, because developing a healthy corporate culture is a particular focus for CompanyA,itknowsthatawell-designedsetofperformancemetricshelpsidentifyimprovement opportunities.
Pilot Experiences
Company A uses a detailed set of operational performance metrics. The definitions ofsomeofthesemetrics,however,differfromthoseusedintheWave1Pilot.ThesedifferencesrequiredCompanyAtoperformadditionalworksoitcouldreportitsdatain the Wave 1 Pilot standardized format.
MetricsreportedbyCompanyA(andallWave1Pilotparticipants)were:
f Data mined retrospectively from the previous 12-month period. f Measured prospectively for the current 3-month period. f Evaluatedatanoperational/siteleveland(whererequired)aggregated
by product/formula. f ReportedtoMcKinseyona“goodenough”basis;datawasnotsubjectedtotherigorousreviewandcheckingrequiredforofficialsubmissiontoFDA.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 81
Appendix 4
Understanding Supply Chain Complexity
CompanyAoperationsinvolvesignificantsupplychaincomplexity,asshownin Figure A1.
Figure A1: Supply Chain Complexity
To patient/customer
To patient/customer
Filling in plant Pack to brown case
Manipulated on another line Manipulated
for local customization
Ship tocustomerWHSE
CMO
Filling in plant Pack to brown case
Manipulated on another line Manipulated
for local customization
Ship tocustomerWHSE
CMO
United States
CentralAmerica
WHSE:WarehouseCustomer: Pharmacy, grocery chain or consumer store
SupplychainsfortwodifferentformulasareshowninFigureA1:
f A red formula (red star) is a bulk drug product made in a manufacturing facility in Country 1 (United States).
f Ayellowformula(yellowstar)isabulkdrugproductmadeinbothCountry1(United States) and Country 2 (Central America).
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 82
Appendix 4
Asthefigureshows,packaging,labelling,secondarypackagingandcustomizingoperations add significant complexity:
f Theyellowformulacanbetransportedinbulktoacontractmanufacturingorganization (CMO) for primary packing, secondary (outer) packaging, and in some cases customization.
f TheyellowformulacouldalsobesenttooneofCompanyA’sexistingplantsineithercountrytobefilled,packaged(primary),dispositioned,andsenttothe customer.
f The primary package could also be moved to another line for manipulation or customization.
f Addingsecondarypackagingintoanoutercartonand/orbrowncasecouldalso include being shipped to a contract manufacturer to be customized.
f Customization can be very complex so that for example, a 24 pack case could consistofmanyproduct/packvariants–24differentformulas,2packsof12 formulas, 6 packs of 4 formulas, or 8 packs of 3 formulas
f A product could be shipped from a location in Country 2 to a location in Country1tobemoveddownthecustomersupplychain.
f Eachprimarypackagehasalotcodeandexpirydatetoallowfulltraceability.
ThiscomplexityrelatesdirectlytohowdatawasreportedbyCompanyAintheWave 1 Pilot: Because the back end of the supply chain includes variation-based choices of the final disposition point (i.e. data at the consumer pack level depends onwherethefinaldispositionpointinthesupplychainisallocated),plant-leveland/oraggregated formula-level metrics that are normalized on a per-pack basis can be affected. Understanding this is critical to ensuring that metrics are captured and reported accurately.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 83
Appendix 4
Supply Chain Complexity in Quality Metric Reporting: Challenges
Given the potential range of primary packs that a secondary (dispositioned) pack might contain for Company A, allocating a specific complaint to a particular product and/or supply chain is particularly difficult.
Assigningconsumercomplaintswoulddependonwherethefinaldispositionwasdesignatedandwheretheconsumerpurchasedthe24-,12-,orsix-packcase.
For these and other reasons, complaint rate results in the Wave 1 Pilot are normalized for a product based on the dispositioned lot.
Other Product-Based Complexity Challenges
In addition to supply chain complexity, Company A also has product line complexity. Whileitisrelativelyeasytogetinformationandqualitymetricdataonproductfamiliesprepared using a base formula differentiated by flavor, for example, this becomes much morecomplicatedwhentherearemultipleformulaswithinonefamily.
CompanyA’sannualproductqualityreviews(APQRs)arebracketedbyformulafamiliesthathaveacommonbase.Onebaseformulawithaparticularflavorcanbepacked into many secondary-pack variations. This makes assigning product-based metrics extremely difficult.
Disaggregatingqualitymetricdata(complaintsrate,forexample)toproductsattheuniqueformulalevelortoaparticularmanufacturingsiteinasupplychainisextremelychallenging;existinginformationtechnology(IT)systemsatCompanyAdonotcurrently facilitate this.
Conclusion
Company A has a large product range and very complex supply chains. This makes assigning metric data to a product level extremely difficult and time-consuming. Changing IT systems to a standardized set of metrics that could produce product- leveldatawouldrequiresignificantinvestment.
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 84
Appendix 5 Detailed Analysis of Data and Relationships for Each Individual MetricLot acceptance rate
Lot acceptance rate defined as difference between 100% and ratio of rejects vs. all dispositioned lots
Effort difficulty
Preferred frequency of reporting
Comments
1.7 2.0 2.1 1.7
2.0 2.3 2.0 1.7
Lots dispositioned
7
13 14
6
43
9
4
95 4
2
Lots dispositioned
Lots rejected
Overall
Difficulty for other components Allocation between components
MONTHLY
Data pulling was not really hard but time consuming (manual pull) (few comments)
We would like to have an option to nclude partial batch rejections
Time consumed [h]2
Lot acceptance rate correlates with: Culture Deviations recurrence Critical complaints Rework
Rejects difficulty1
Drug subst. OTC Gx Rx
1 Onascalefrom1-easiestto4-mostdifficult 2 Retrospective+current
Lot acceptance rate%releasedoutofallfinallydispositionedlotsLot acceptance rate
98.9
87.9
97.0
100.0 99.7
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 85
Appendix 5
Confirmed OOS – product
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
Confirmed OOS – product defined as (total confirmed OOS – confirmed stability OOS – cofirmed RM OOS)/(total lots tested – stability lots tested – RM lots tested)
1.92.5
1.9 1.720
23
915
Confirmed OOS – product correlates with: Total complaints (lag) Critical complaints (lag)
Collecting separately release OOS and lots tested would simplify and speed up the process (2 components instead of 6 for KPI) (McK.)
Numbers for lots tested get huge for multi-product plant or when lots of water is tested (few comments)
Would like more clarity which lots to include in lots tested (e.g. placebo, micro samples)
For OOS could be specifed‚ count when investigation closed’ as only then it can be‚ confirmed’
Total confirmed OOS difficulty1
2.42.12.42.11.8
2.52.3
2.5
2.32.0 1.81.81.72.21.72.32.02.12.02.0
RM lo
ts
test
ed
Stab
ility
lots
test
ed
Tota
l lot
s te
sted
Con
firm
ed
RM O
OS
Con
firm
ed
stab
ility
OO
S
4.61.8
7.0
1.41.2
4.0 1.53.84.64.6
7.5
1.40.9
2.3
1.21.0 2.81.4
4.6
1.21.14.2
RM lo
ts
test
ed
1.5
Tota
l lot
s te
sted
Tota
l lot
s te
sted
Con
firm
ed
RM O
OS
Con
firm
ed
stab
ility
OO
S
Tota
l co
nfirm
ed
OO
S
2.4
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Confirmed RM OOS – productNo unit provided
Confirmed OOS – product 150.6
0.1 0
0.4
0 0
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 86
Appendix 5
Confirmed OOS – stability
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
Confirmed OOS – stability defined as ratio of confirmed OOS for stability to stability lots tested
1.8 2.0 1.7 2.0
2.1 2.3 1.8 2.0
Lots dispositioned
4
8
43
2.11.4 1.9
6.0
2.02.2 1.41.1
Lots tested stability
Confirmed OOS stability
Data for stability stored separately/pulled manually (few comments)
At least for sites with only few products it makes little sense to collect monthly (GMP requires 1 batch/year/ product)
Confirmed OOS – stability correlates with: -
Confirmed OOS stability difficulty1
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Confirmed OOS – stabilityPer 000’ stability lots tested
0 0
7.1
250.0
2.3
25.3
Bio
API
Steriles
Solids
Other
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 87
Appendix 5
Deviations rate
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
Deviations rate defined as ratio of deviations to lots dispositioned
1.62.3 1.9 1.5
2.0 2.3 2.0 1.7
Lots dispositioned
6
1310
6
52
94
10
14
1
Lots dispositioned
# of deviations
Deviations rate correlates with: Critical complaints
# of deviations difficulty1 Deviation def. is very broad and includes almost everything, should be narrowed, also adding OOS here confuses the picture Need more clarity if e.g. RM or buffer deviations are to be counted Issues with data pulling (e.g. separate systems for OOS and deviations and need for aggregation) (few comments)
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Deviations ratePer 000’ lots dispositioned
13
15,667
256 128
683
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 88
Appendix 5
Rework rate
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
Rework rate defined as ratio of reworked and reprocessed lots vs. (lots dispositioned minus lots rejected)
1.9 1.52.0 1.7
2.01.7 2.32.0 2.02.1 1.71.7
Lots rejected Lots dispositioned
7
18 19 17
421
945
116
1
953
Rework rate correlates with: Lot acceptance rate
Lots reworked difficulty1We do not permit any rework at all (few comments)
Need more clarity what rework or reprocess mean
Difficulty for other components Allocation between components
Lots rejected
Lots reworked
Lots dispositioned
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Rework rate%lotsdispostioned
2.43
1.20
0.48
00
25.41
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 89
Appendix 5
Confirmed OOS – raw materials
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
Confirmed OOS – raw materials defined as ratio of confirmed OOS for raw materials to raw materials lots tested
2.1 2.3 2.2 2.0
2.4 2.51.8 2.3
Lots tested - RM
4
85
4
2.90.9 1.9
6.0
2.72.3 2.51.0
Confirmed OOS - RM
Lots tested - RM
Confirmed OOS – raw materials correlates with: -
Confirmed OOS RM difficulty2 sites did not supply it Need more clarity what to include (e.g. which EM, why water is in and other EM not neccessarily) (few comments)
Split between technologies and/or pharma/cosmetics businesses difficult (few comments)
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Confirmed OOS – raw materialsPer000’RM/PMlotstested
40.20
0
2.24
0.40
5.22
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 90
Appendix 5
Unconfirmed OOS
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
Unconfirmed OOS defined as ratio of unconfirmed OOS to total lots tested
2.1 2.31.6 1.9
2.4 2.51.7 2.1
Lots tested
10
37
7 12
72
1720
349
2
Lots tested Unconfirmed OOS
Unconfirmed OOS correlates with: -
Unconfirmed OOS difficulty
Data is automatically available together with confirmed OOS (few comments)
There was some manual aggregation needed due to scope (few comments)
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Unconfirmed OOSPer 000’ lots tested
3.0
869.3
8.8
0 0.8
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 91
Appendix 5
Recall events US
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
Recall events US
1.2 1.5 1.3 1.30.5
1.30.7 0.7
Recall events US correlates with: Deviations rate Critical complaints
This is non-applicable for some drug substance sites
Time was short as we had no recalls, not sure if this reflects situation with recalls
We should not limit ourselves to US market
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Recall events US# of events
1
000
3
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 92
Appendix 5
Recall events class I and II US
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
Recall events US class I and II US
Recall events US class I and II US correlates with: Not tested
1.2 1.3 1.3 1.30.3
1.40.7 0.5
This is non-applicable for some drug substance sites
Time was short as we had no recalls, not sure if this reflects situation with recalls
We should not limit ourselves to US market
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Recall events class I and II US# of events
3
0000
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 93
Appendix 5
Recalled lots US
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
Recalled lots US
Recalled lots US correlates with: Not tested
1.2 1.3 1.4 1.40.3
1.40.9 0.5
This is non-applicable for some drug substance sites
Time was short as we had no recalls, not sure if this reflects situation with recalls
We should not limit ourselves to US market
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Recalled lots US# of lots
4
0000
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 94
Appendix 5
Total complaints rate
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
OTC Gx Rx
Total complaints rate defined as ratio of total complaints to total packs released
1.8 2.0 2.0
1.82.5 2.1
Packs released
1220
34
7513
7
29
5
Packs released
Complaints
Total complaints rate correlates with: Action limit excursions OOS product (lag)
Total complaints difficulty1 1 plant did not manage to collect packs data Had trouble to report packs due to different system set-up Difficult to split between technologies Time consuming due to many products (few comments)
Some complaints cannot be allocated to product at all Need more clarity – e.g. whether include market only or also internal complaints,if bulk complaint should be reported bybulk of packaging site, how to compare complaints on bulk with packs (few comments)
Little value added in that, confirmed complaints would be better
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Total complaints ratePer million packs
26
86
14
1,078
1
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 95
Appendix 5
Critical complaints rate
Effort difficulty
Preferred frequency of reporting
Comments
Overall
MONTHLY
Time consumed [h]2
OTC Gx Rx
1.82.3 2.1
1.82.5 2.1
Packs released
1020
231 1 plant did not manage to collect
packs data Had trouble to report packs due to different system set-up Difficult to split between technologies (few comments)Time consuming due to many products (few comments) Some complaints cannot be allocated to product at all
Difficulty for other components Allocation between components
Critical complaints rate defined as ratio of total complaints to total packs released
73
137
18
5
Packs released
Critical complaints
Critical complaints rate correlates with: Deviations rate Lot acceptance rate OOS product (with lag)
Critical complaints difficulty
If we have a complaint on bulk how do we report corresponding packs? Had to manually assess each complaint for criticality Little value added in that, confirmed complaints would be better (few comments)
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Critical complaints ratePer million packs
2.0
0.7
0.10
161.8
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 96
Appendix 5
APQR on time
Effort difficulty
Preferred frequency of reporting
Comments Overall
ANNUALLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
APQR on time defined as ratio of APQR on time to all APQR closed in reporting period
1.42.0
1.1 1.3
1.42.0
1.3 1.4
Products subject to APQR
1.0
3.6
2.01.1
0.40.6
1.91.71.01.0 0.50.5
Products subject to APQR
APQR on time
We group our products for APQR (few comments)
Period may need to be adjusted for this point, products evaluated are different than ones produced in the period
APQR on time correlates with: CAPAs effective
APQR on time difficulty
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
APQR on timePercent
0
100.0
42.9
100.0 100.0
90.9
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 97
Appendix 5
CAPA effectiveness
Effort difficulty
Preferred frequency of reporting
Comments Overall
QUARTERLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
CAPA effectiveness defined as ratio of CAPAs evaluated as effective to all CAPAs evaluated for effectiveness in the period
2.12.7 3.0
1.8
1.8 2.03.0
2.1
CAPAs with effectiveness check
5
20
2 4
23 4
16
11 22
CAPAs with effectiveness check
CAPAs effective
27% of plants did not collect enough data for this metric Manual extraction of data from e.g text files (few comments) We evaluate group of CAPAs as a whole and not single CAPAs Need clarity how to include partially effective CAPA Recurrence of the "quality issue" is secondary. CAPA target a specific root cause and to be effective should address the given root cause which subsequently prevents recurrence of the "quality issue"
CAPA effectiveness correlates with: APQR on time
Effective CAPA difficulty
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
CAPA effectivenessPercent
Bio
API
Steriles
Solids
Other 1
Lab 0
86.5
35.3
93.6 97.7
100.0
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 98
Appendix 5
Recurring deviations
Effort difficulty
Preferred frequency of reporting
Comments Overall
MONTHLY
Time consumed [h]2
Drug subst. OTC Gx Rx
1
Recurring deviations defined as ratio of recurring deviations (with 1 year look-back period) to all deviations closed
2.63.8
2.7 2.3
1.62.3 1.9 1.5
Number of deviations
49
16
5
12 45 3
13
23
Number of deviations
Recurring deviations
Recurring deviations correlates with: Lot acceptance Recalls Culture
Recurring deviations difficulty 15% of sites did not report it at all Deviation def. is very broad and includes almost everything, should be narrowed, also adding OOS here confuses the picture Need more clarity if e.g. RM or buffer deviations are to be counted Issues with data pulling (e.g. separate systems for OOS and deviations and need for aggregation, manual checking) (few comments) recurring Deviation should be limited to the same deviation (same specific departure from approved process or instructions) and independent of "root cause". The given definition of "deviation" is so broad many things that meet that definition likely do not require root cause analysis
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Recurring deviationsPercent
13.5
3.90
62.3
25.4
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 99
Appendix 5
Media fills successful
Effort difficulty
Preferred frequency of reporting
Overall
ANNUALLY
Time consumed [h]2
Rx
1
Media fills successful defined as media fills successful to all media fills
1.8
1.8
Total media fills
1.1
0.40.7
Media fills successful Total media fills
Media fills successful correlates with:
Media fills successful difficulty
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Media fills successfulPercent
100.0100.0
Bio
API
Steriles
Solids
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 100
Appendix 5
Lots rejected due to action limit excursions
Effort difficulty
Preferred frequency of reporting
Overall
QUARTERLY
Time consumed [h]2
Rx
1
Lots rejected due to action limit excursions defined as ratio of lots rejected due to action limit excursions to total sterile lots dispositioned
2.5
1.9
Total sterile lots
dispositioned
3.5
1.81.8
Total sterile lots
dispositioned
Sterile lots rejected for limit exc.
Lots rejected due to action limit excursions correlates with: -
Sterile lots rejected for action limit - difficulty
Difficulty for other components Allocation between components
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Lots rejected due to action limit excursionsPercent
0.1
0
0.5
0
Bio
API
Steriles
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
Report from ISPE Quality Metrics Pilot Program / Wave 1 – June 2015 101
Appendix 5
Investigations related to action limit excursions
Effort difficulty
Preferred frequency of reporting
Overall
MONTHLY
Time consumed [h]2
Rx
1
2.8
1.9
Total sterile lots
dispositioned
3.6
1.81.8
Total sterile lots
dispositioned
Difficulty for other components Allocation between components
Investigations related to action limit excursions defined as ratio of lots with action limit excursions to total sterile lots dispositioned
Lots with action limit excursions
Investigations related to action limit excursions correlates with: Total complaints Culture (capabilities)
Investigations difficulty
1 Onascalefrom1-easiestto4-mostdifficult2 Retrospective+current
Investigations related to action limit excursionsPercent
11.7
00.8
2.8
15.4
Bio
API
Steriles
Other 1
Lab
1 OtherincludesCreams,LiquidsandOther
© 2015 International Society for Pharmaceutical Engineering
All rights reserved