WHO in vitro diagnostic prequalification and post-market surveillance for new applicants: a detailed overview
Helena Ardura, Mercedes Perez and Anita SandsWHO PQ Team – Diagnostics AssessmentEssential Medicines and Health Products
1Copenhagen, Denmark 18-21 September 2017
The importance of IVDs
2Copenhagen, Denmark 18-21 September 2017
• Every year, hundreds of millions of US dollars’ worth of IVDs are purchased for distribution in resource-limited countries.
• IVDs play a critical role in ensuring blood safety, surveillance, diagnosis and treatment initiation and monitoring.
• IVDs save lives, reduce suffering and improve health, but only when they are of good quality, safe, effective, available, affordable, acceptable and properly used.
IVDs QA activities within WHO
3Copenhagen, Denmark 18-21 September 2017
• WHO has been assessing IVDs performance and operational characteristics since 1988 :
• HIV assays since 1988
• Hepatitis B assays since 2000
• Hepatitis C assays since 2000
• Syphilis assays since 2001
• Chagas assays since 2002
• Malaria assays since 2002
• CD4 technologies ad-hoc in 1996 & 2003
Trends in IVDs
4Copenhagen, Denmark 18-21 September 2017
• Globalised industry sectors with outsourced production
• Rapid emergence of new technologies • Increasing expectations on quality, safety and
performance• Increasing workload for regulators• Easy to operate tests/methods facilitate near
patient testing, hard-to-reach populations, non-lab environments
WHO's response: PQ in a regulatory framework
5Copenhagen, Denmark 18-21 September 2017
Pre-market Post-
market
Performance Quality Safety
Pre-2008
2008 - 2014
Post-2014
Changes
Performance Quality Safety
Reliance
6Copenhagen, Denmark 18-21 September 2017
• The aim of PQDx is to promote and facilitate access to safe, appropriate and affordable IVDs of good quality
• Focus is placed on IVDs for priority diseases and their suitability for use in resource-limited settings
PQDx: aim, scope and impactHIV
Malaria
Hepatitis C
Hepatitis B
HPV
G6PD
Cholera*
*As of January 2018
7Copenhagen, Denmark 18-21 September 2017
• The findings of PQDx generate independent technical information on safety, quality and performance of IVDs, principally used by other UN agencies, WHO Member States and other interested organizations.
• The PQDx status, in conjunction with other procurement criteria, is used by UN agencies, WHO Member States and other interested organizations to guide their procurement of IVDs.
PQDx: aim, scope and impactHIV
Malaria
Hepatitis C
Hepatitis B
HPV
G6PD
Cholera*
*As of January 2018
PQDx components
PQDx undertakes a comprehensive assessment of
individual IVDs through a standardized procedure aimed at
determining if the product meets WHO prequalification
requirements.
• The prequalification assessment process includes
three components:
Review of a product dossier
Performance evaluation
Manufacturing site(s)
inspection
Reference documents
PQDx is aligned with best international practice for IVDs
• ISO (and EN) standards
• Global Harmonization Task Force/International Medical
Device Regulators Forum (GHTF/IMDRF) guidance
• CLSI guidance
• Requirements of mature national regulatory authorities
including: FDA, EU, TGA, Health Canada, Japanese
Ministry of Health, Labour and Welfare
What does PQ do differently to regulators?
Requirements are based on the same set of standards – PQ is aligned with internationally accepted practice
BUT
PQ review is of aspects of particular relevance for resource-limited settings
WHO prequalification: Full assessment
Dossier review Site inspection
Dossier incomplete
Laboratory evaluation
Prequalification decision
Dossier complete
Dossier screening
Priority product
Yes
No
Pre-submission form
New
pathway
Maintenance of PQ status(PMS, changes, commitments, annual report, re-inspection)
Resources optimization: full Vs abridged assessment
Intention: harmonization initiatives, leveraging existing evidence and avoiding duplication of efforts:
• Comparison of requirements with existing mechanisms
• Benefit of existing evidence but independent decisions
abridged PQ assessment
Determining eligibility for abridged assessment
• Submission and review of WHO pre-submission form
• Evidence of prior regulatory review of the product or a
similar product manufactured by the same manufacturer
• Products will not be eligible for abridged assessment
based on anticipated stringent regulatory review
• Contact WHO to discuss before submitting
pre-submission form
Prequalification of IVDs Programme
Pre-submission form Pre-submission form
Dossier review Site inspection Laboratory evaluation
Dossier incompleteDossier incomplete
Prequalification decision
Dossier completeDossier complete
Dossier screeningDossier screening
Priority productPriority product
YesYes
NoNo
Pre-submission formPre-submission form
Abridged site
inspection
Laboratory evaluation
Prequalification decision
Yes Yes
Full PQ assessment
Full PQ assessment
NoNo
YesYes
Priority productPriority product
NoNo
Decision on Abridged PQ assessment
Decision on Abridged PQ assessment
Full prequalification assessment Abridged prequalification assessment
PQDx components
The prequalification assessment process includes 3 components:
Review of a product dossier
Performance evaluation
Manufacturing site(s) inspection
Purpose of the product dossier
• The dossier contains a subset of the technical
documentation held by the manufacturer • to demonstrate that the IVD to which it applies conforms to the
“Essential Principles of Safety and Performance of Medical Devices” as defined by GHTF
• The dossier reflects the status of the IVD at a
particular moment in time
Purpose of the dossier
• It should also provide sufficient information to inform the
PQ Inspection team regarding:• Sites responsible for design and manufacture to enable planning of
inspection/s
• Information regarding the maturity of the manufacturer’s QMS
• It should provide sufficient information to determine the
regulatory version submitted to PQ and to ensure the
data in the dossier is relevant to this version
• It should demonstrate that the manufacturer has
considered the safety and performance in WHO Member
States.
PQDx components
Performance evaluation
Manufacturing site(s) inspection
Requirements for Manufacturing site inspection
� Fully implemented quality management system (design &
development, manufacturing including quality control,
storage, distribution)
� Risk management to meet ISO 14971:2007
� Robustness of the Product
� Products undergoing prequalification have to be in routine
manufacturing
� Sufficient capacity to ensure reliable delivery
PQDx requirements prior to inspection
� Transfer from R&D to production completed
� Established and evaluated suppliers
� Validated processes (acceptance ranges determined, in-
process controls established)
� Trained personnel (requirements determined, training plan,
records)
� Established "out-of-specification" process
� Batch manufacturing records established (include all
manufacturing information, full traceability of material and
equipment)
Quality Management System
Fully implemented quality management system (design &
development, manufacturing including quality control,
storage, distribution)
• Meets ISO 13485:2003 requirements
• Competence of personnel
• Work environment
• Quality control processes, quality control plan
established
• Storage conditions, temperature and humidity,
components and kit, real time data required
Risk management
ISO 14971:2007
• Risk Management Plan – Policy
• Risk analysis
• Risk evaluation
• Risk control
• Evaluation of residual risk
• Risk management report
• Production and post production
information feedback
• Risk management file
• Evaluation of readiness for inspection (Stage 1)
• Desktop review of QMS documentation
• Stage 1 inspection (1 inspector day to inspect QMS
status, facility, staff competencies, critical suppliers incl.
outsourced activities, internal audit and management
commitment / review)
• Onsite Inspection (Stage 2)
• Follow up onsite inspection
• Re-Inspection (risk based, after 3-5 years)
Inspection
25Copenhagen, Denmark 18-21 September 2017
PQDx components
Performance evaluation
Performance evaluation
26Copenhagen, Denmark 18-21 September 2017
• Independent verification of the performance of IVDs submitted for prequalification assessment.
• Assays are challenged with a focus on their use in resource-limited settings and in the context of WHO guidelines (SRA review has different priorities based on local populations and product use)
• The dataset obtained complements the verification and validation data submitted by the manufacturer in the product dossier
• Currently takes place in a WHO Collaborating Centre (CC) and/or a site otherwise designated by WHO
Performance evaluation
27Copenhagen, Denmark 18-21 September 2017
Dossier review Site inspection
Dossier incomplete
Laboratory evaluation
Prequalification decision
Dossier complete
Dossier screening
Priority product
Yes
No
Pre-submission form
New
pathway
Performance evaluation pathways
28Copenhagen, Denmark 18-21 September 2017
Option 1: Performance evaluation coordinated by WHO at an
earlier stage of the prequalification process:
• The performance evaluation will be scheduled by
WHO as soon as the product is designated as
meeting WHO prioritization criteria.
Performance evaluation pathways
29Copenhagen, Denmark 18-21 September 2017
• Option 2: Performance evaluation commissioned by the manufacturer and carried out at a Prequalification Evaluating Laboratory listed by WHO
• The manufacturer selects a laboratory from the list of WHO Prequalification Evaluating Laboratories
• The manufacturer will bear the cost of the evaluation and be responsible for coordinating it directly with the laboratory
• The selected evaluating laboratory will inform PQ as soon as an evaluation has been commissioned by a manufacturer
Performance evaluation
30Copenhagen, Denmark 18-21 September 2017
Preparation
• Acceptance of Pre-submission form and prioritization
• Identification of WHO Collaborating Centres/designated site
• Protocol is sent to the designated site(s) and the manufacturer
• Ethical approval by the WHO Collaborating Centre
• Delivery of kits to the WHO Collaborating Centre
Testing
• WHO verify the Instructions for Use
• Manufacturer demonstration at the WHO Collaborating Centre/designated site
• Testing using approved protocol and manufacturer
• Draft laboratory evaluation report sent to WHO
Report
• WHO review the submitted draft report and submit to the manufacturer
• Manufacturer submit comments to WHO within 30 days
• WHO send the final report to the manufacturer
Evaluation of molecular technologies (NAT): collaboration with different partners
31Copenhagen, Denmark 18-21 September 2017
• HIV NAT qualitative (EID) and quantitative (VL) assays: Collaboration with ILB, CDC Atlanta:• In 2014, WHO and PEPFAR decided to align QA
mechanisms for procurement eligibility.
• Goal of alignment: to reduce duplication of effort for each organization and for the manufacturers, leverage each others resources for laboratory evaluation, and to create one list of "approved" products.
• Other partners involved in HPV and HCV evaluations
32Copenhagen, Denmark 18-21 September 2017
Performance evaluations on molecular technologies are divided into:
– Analytical performance studies: precision, LOD, carry-over, linearity,
subtype detection.
– Clinical performance studies: sensitivity, specificity, misclassification rate,
bias.
– Invalid rates
• Panels may be composed of retrospectively or prospectively collected
specimens
– Adhere to manufacturer’s instructions and claims
• Analytical and clinical performance studies may be conducted by different
sites
Molecular technologies
33Copenhagen, Denmark 18-21 September 2017
Performance evaluations on molecular technologies are divided into:
– Analytical performance studies: precision
– Clinical performance studies: classification at clinically relevant levels.
– Invalid rates
• Panels are prospectively collected as they require fresh whole blood
– Adhere to manufacturer’s instructions and claims
CD4 Technologies
34Copenhagen, Denmark 18-21 September 2017
• Serology evaluations are performed on archived panels from different geographical regions.
• Performance characteristics include:
• sensitivity and specificity
• limit of detection
• inter-reader variability (RDTs)
• seroconversion sensitivity
• antigen detection if applicable
• genotype detection
• lot-to-lot variation
Serology evaluations: HIV, HIV/TP, HBsAg, HCV
Operational Characteristics
35Copenhagen, Denmark 18-21 September 2017
Operational characteristics are assessed with a focus on the IVDs’ use in resource-limited settings. A standard appraisal sheet is used by technicians.
• Time to first result
• Infrastructure requirements (footprint of instrument, environmental requirements, electrical needs etc.)
• Waste disposal and biosafety
• Specimen and reagent storage and transport
• Materials required but not provided
• Level of skill required
• Calibration and maintenance needs
• Data management
• Instructions for use
• Etc.
Prequalification decision
36
Prequalification decision
Prequalification: decision
37Copenhagen, Denmark 18-21 September 2017
• Final prequalification outcome depends on: • Results of dossier assessment and acceptance of action plan
• Results of inspection(s) and acceptance of action plan
• No level 5 nonconformities outstanding for either dossier or for inspection
• Meeting the acceptance criteria for the laboratory evaluation
• WHO PQDx Public Report is posted on WHO website and product is added to the list of WHO prequalified products
• Product is then eligible for WHO and UN procurement
Post - Prequalification Activities
38Copenhagen, Denmark 18-21 September 2017
Maintenance of Prequalification Status
39Copenhagen, Denmark 18-21 September 2017
Manufacturers are required to notify WHO of planned changes made to any PQed product.
• Clear guidance available as to what changes must be reported
• Not all changes will be charged an assessment fee
• Early notification to PQ team advised
Changes to prequalified products
40Copenhagen, Denmark 18-21 September 2017
Every year, manufacturers are
required to report to WHO:
• Sales data
• Number of complaints
• Number of field safety
corrective actions
Annual reporting
41Copenhagen, Denmark 18-21 September 2017
• Manufacturers are required to report:• Any serious adverse event to the relevant
NRA within their respective timelines, and to WHO within 10 days.
• Any moderate adverse event or any change in the trend of mild adverse events to the relevant NRA within their respective timelines, and to WHO within 30 days.
• All complaints (both administrative and technical including serious, moderate and mild adverse events) to the relevant NRA, and to WHO as a periodic summary report, annually.
Post-Market Surveillance
42Copenhagen, Denmark 18-21 September 2017
• Any non/critical dossier or inspection “non-
conformities” must be resolved within the
agreed timelines
• Implementation will be reviewed at the next
inspection
Commitments to Prequalification
43Copenhagen, Denmark 18-21 September 2017
• Ongoing compliance with the requirements of
WHO PQ, including ISO 13485
• Timing will depend on
• Number and nature of non-conformities identified in
initial inspections/dossier
• Number of complaints
• Number and nature of changes
• Production figures
Re-inspection
Guidance for manufacturers
• WHO has placed an
increased emphasis
on the creation of
better guidance.
Guidance for manufacturers applying to PQDx
� Sample dossiers
� Technical Guidance Series
� Technical Specification Series
http://www.who.int/diagnostics_laboratory/g
uidance/en/
How to ensure your dossier meets PQ expectations
46Copenhagen, Denmark 18-21 September 2017
Sample dossiers
• Fictitious IVDs
• Provide examples of formatting and reporting details required
• Also examples of how to complete an Essential Principles checklist
• Good examples of risk assessment
Technical Guidance Series
47Copenhagen, Denmark 18-21 September 2017
• The Technical Guidance Series (WHO PQDx TGS) is produced for manufacturers interested in WHO prequalification of their IVD.
• The series is intended to help manufacturers in meeting prequalification requirements by making process more transparent and improving quality of submissions
Goal: to continue to create TGS in response to needs identified primarily in the dossier assessment phase of PQ.
Technical Guidance Series (TGS)
48Copenhagen, Denmark 18-21 September 2017
Provide detailed guidance on a specific aspect related to IVD performance.
• Currently published or in draft:• TGS 1 Standards applicable to the WHO Prequalification of
in vitro diagnostics• TGS 2 Establishing stability of an in vitro diagnostic for WHO
Prequalification• TGS 3 Principles of performance studies• TGS 4 Test method validation of in vitro diagnostic medical
devices• TGS 5 Panels for quality assurance and quality control of in
vitro diagnostic medical devices• TGS 6 Instructions for use
Technical Specifications Series (TSS)
49Copenhagen, Denmark 18-21 September 2017
• Summarize minimum performance requirements for WHO prequalification, to establish:
• appropriate performance evaluation and re-evaluation criteria,
• appropriate reference methods and reference materials.
• Specific requirements tailored to types of infections, conditions, etc.
• Clarify requirements for:• Manufacturers
• Assessors
Technical Specifications Series (TSS)
50Copenhagen, Denmark 18-21 September 2017
• Currently published or in preparation• TSS1: HIV RDTs (published)
• TSS2: G6PD RDTs (published)
• TSS3: Malaria RDTs (published)
• TSS4: HPV NATs (published)
• Planned• HIV NAT – quantitative & qualitative
• HCV NAT - quantitative & qualitative
• HBsAg, HBc RDTs and EIAs
• HCV RDTs and EIAs
• POC CD4
• HIV/Syphilis RDTs
• QC for RDTs
• Accessories
Q&A
51Copenhagen, Denmark 18-21 September 2017
WHO normative guidance on PMS
� Stakeholders’ roles/responsibilities
– Manufacturers
– End-users
– National regulatory authorities
– National reference laboratories
� Template forms
– IVD complaint report
– Manufacturer investigation report
– Field safety corrective action report
WHO post-market surveillance of IVDs
Reactive PMS
Evaluation of EQA/QC dataEvaluation of EQA/QC data
Pre-distribution Pre-distribution
Possible issuance of Field Safety Notice
Post-distribution Post-distribution
ComplaintComplaint
Possible Field Safety Corrective Action
Possible Field Safety Corrective Action
Lot verification testingLot verification testing
Proactive PMS
Any class of IVDAny class of IVD
Examples of complaints related to IVDs Serious – death or serious injury of patient, user or other person, false negative results
An individual is diagnosed as HIV+ after transfusion of a
blood product. The blood donation was screened as
negative by an HIV-1/2 RDT.
A HIV+ individual reporting for ART initiation was re-tested
to confirm their HIV diagnosis. The re-testing results are
HIV negative.
A healthcare worker operating a small instrument-based
test at POC couldn't remove a cartridge and inserted a
knife into the instrument and was electrocuted, resulting in
death.
Examples of complaints related to IVDs
Examples of complaints related to IVDs
Moderate - Anomalies that lead to a higher than expected invalid/error rate, unreturnable or inconclusive results, false positive results
The invalid rate for an RDT exceeds 5%, for any reason.
Higher than usual background which may or may not
obscure reading window and prevent reading of results.
Greater than expected discrepant rate between assay 1 and
assay 2, suspected reduced specificity of assay 2.
Mild - Deficiency found prior to use, expected and foreseeable side effects
The packaging of a single use device is labelled with the
caution 'do not use if the packaging is opened or damaged'.
Prior to use, obvious damage is observed and the device is
not used.
A component labelled as lyophilized is found to be fluid, this
is discovered by the user prior to use. Entire test kit must be
discarded
Desiccant has changed colour/density. Device is discarded
and a new device used.
Examples of complaints related to IVDs
WHO receipt of IVD complaints
• Complaints from manufacturer
– Any serious event must be reported
to WHO within 10 days
– Any moderate event or change in
trend of mild events must be
reported to WHO within 30 days
– All complaints must be reported
annually in summary form
• Complaints from end-users
– As soon as you become aware
Complaint notified using WHO IVD complaint form Complaint notified using WHO IVD complaint form
WHO reviews information contained in complaint form
WHO reviews information contained in complaint form
WHO forwards complaint form
to manufacturer, if received from
end-user
WHO forwards complaint form
to manufacturer, if received from
end-user
WHO opens complaint file, assigns reference number (COMP-xxxx)
WHO opens complaint file, assigns reference number (COMP-xxxx)
WHO’s expected response from manufacturer
• Root cause analysis
– How/why did this happen
• Analysis regarding related areas
– Is this same issue impacting/occurring elsewhere
• Correction with completion dates
– Fix now
• Corrective action, if applicable, with planned completion
dates
– To prevent recurrence
WHO review of manufacturer investigation
• WHO reviews investigation
report and supporting
documentary evidence within
7 days of receipt
• May request for clarification by
official letter
• Aim is to evaluate if complaint
handling procedure has been
implemented properly
Manufacturer submits initial investigation report
Manufacturer submits initial investigation report
WHO reviews investigation report and prepares letter with any requests
for clarification
WHO reviews investigation report and prepares letter with any requests
for clarification
WHO may request
additional information from
end-user
WHO may request
additional information from
end-user
WHO reviews follow-up investigation report to see if queries have been
answered
WHO reviews follow-up investigation report to see if queries have been
answered
Final investigation report is received at end of enquiry or when FSCA is
complete
Final investigation report is received at end of enquiry or when FSCA is
complete
WHO review of manufacturer investigation cont’d
• For each complaint, the following activities may be required
– Testing and inspection of retained samples from affected lots;
• WHO will review the testing panel used in the investigation, i.e. specific
investigative panel or usual QC lot release panel
• WHO will review acceptance criteria for lot testing conducted;
– Review of stability of product (all components) at different
temperatures or humidity (claimed shelf life, in-use stability,
shipping stability);
– Complete risk assessment report;
– Update risk management file.
• Risk management in the intended use setting should guide
the investigation
Root cause analysis
• RCA using methodology such as fishbone (Ishikawa)
diagram to determine possible causes for an observed
effect (i.e. the problem)
• 6 Ms
– Methods
– Machines (equipment)
– (Man) power
– Materials
– Measurement
– Mother nature
Analysis of related areas
• Why is this important?
• If the root cause is related
to the quality system, then
other products are likely to
be affected
• Case study
– High rate of invalids,
related to certain lots of
nitrocellulose
Correction (fix now)
• If deemed necessary, a correction
may be conducted before the root
cause is determined
• For example, certain lots are
conclusively known to be affected
and will give incorrect results,
decision to recall these lots is a
correction
Corrective action (prevent reoccurrence)
• Actions taken to prevent reoccurrence of problem
• Examples
– Change in standard operating procedure for manufacturing
– Change in composition of QC panel
• WHO will determine:
– If suggested corrective/preventive actions are acceptable;
– If suggested actions for customers are acceptable;
– If risk management update is acceptable.
WHO complaint handling since 2015 (n=67)
• In order of frequency
– False negative
results
– ↑ invalid rate
– False positive
results
– Defective reagent
Type of complaint
False negative
False negative andfalse positiveFalse positive
Falsification
Invalid rate
Mislabelled
Unreturnable result
Defective reagent
Weak reactivity
Software
Erroneous results
Reactive PMS: Field Safety Corrective Action
• A FSCA is triggered when there is an unacceptable increase in risk vs. benefit associated with use of the IVD
• FSCAs may include
• Changes to labelling/IFU
• Recalls/destruction
• Exchange (swap-out)
• End-users will be informed using a Field Safety Notice sent by the manufacturer, via their in-country distributor
Manufacturer submits draft FSN to WHO
Manufacturer submits draft FSN to WHO
WHO reviews in draft FSN and gives go-ahead for dissemination
WHO reviews in draft FSN and gives go-ahead for dissemination
WHO posts FSN on WHO website
for WHO prequalified IVDs
WHO posts FSN on WHO website
for WHO prequalified IVDs
Manufacturer submits FSCA report with subjective measure of FSCA
effectiveness
Manufacturer submits FSCA report with subjective measure of FSCA
effectiveness
Manufacturer sends FSN to all affected customers
Manufacturer sends FSN to all affected customers
Proactive post-market surveillance of IVDs
• Evaluation of EQAS and QC
testing results
– Across sites using the same
assay, same/different lot
• Lot verification testing
– Independent of the
manufacturer, NOT a
substitute for their lot release
testing
– Ensures that only lots
continue to meet established
criteria
– Uses a risk-based approach
Evaluation of EQA/QC dataEvaluation of EQA/QC data
Pre-distribution Pre-distribution
Possible issuance of Field Safety Notice
Post-distribution Post-distribution
Possible Field Safety Corrective ActionPossible Field Safety Corrective Action
Lot verification testingLot verification testing
Proactive PMS
Proactive PMS of IVDs: evaluation of EQA/QC data
• Review data from external
quality assessment schemes
(proficiency testing) and end-
user QC
• Greatest value when there are
many users are of the same
assay
• Useful to report differences in
performance such as this
example of a shift (lot to lot)
Proactive PMS of IVDs: lot testing
• Lot verification by suitably qualified laboratory using SOPs
so that each lot testing event is consistent
• Through physical inspection of packaging, labelling and
instructions for use
– Looking for breaches of packaging that might affect stability
Proactive PMS of IVDs: results of lot testing
• Lot testing panel characterized by an agreed reference
standard
– Positive and negative clinical specimens, diluted specimens to
test analytical sensitivity (LoD)
– Record invalid rates and other anomalies as these are
significant, e.g. incomplete clearing, high background, faint lines
• Lot acceptance criteria must be in place (pass/fail) for both
inspection and testing
Contact us
Contact us by email
By emailing [email protected]
http://www.who.int/diagnostics_laboratory/evaluations/en/
Sign up to our mailing list
Check our website