Embed Size (px)
DESCRIPTION
This presentation identifies a way to use Risk Management to determine the extent and scope of a validation project, including what validation documents are needed and what should be tested. One validation size does not fit all validation projects! Using the Quality/Regulatory Risk and Functionality/Distribution Risk identifies an Overall System Risk. The Overall System Risk and the Type of System Change determine the needed Validation documents. A methodology to identify the extent of validation test scripts is discussed too.
Citation preview
Estimating the Dilbert Way (1 of 2)
Estimating the Dilbert Way (2 of 2)
Agenda
• Brief Overview of Validation and Verification• Do we have to Validate the System?• Identify Overall System Risk
• Identify Type of System Change• Identify Needed Validation Documents• Identify Extent of Test Scripts
• Workshop
DISCLAIMER
• Everything in this presentation represents my opinion and not that of my employer.
• It’s all about RISK!
Validation and Verification
• Verify each part: Test and document results
• Validation = ∑ verified parts
IQ OQ PQ
TMURS FS
QPP QSR
VERIFY
It’s the entire process!
Potential Validation Documents(GAMP 5)
Validation Plan
User Requirements
Functional Specs
Risk Analysis
Design specs
Trace Matrix
IQ Protocol/ Scripts
OQ Protocol/ Scripts
PQ Protocol/ Scripts
Validation Summary
Data Migration (DM)
Key Validation Deliverables• Number of validation documents
– How many?
• Extent of the test scripts– What and how much to test?
vs.
vs.
How Much? FDA: : Use Risk
How Much? QA vs. IT
QA IT & CUSTOMER
VALIDATION
Validation Estimating Workflow
Determine Type of System Change
Determine Quality & Regulatory Risk
Determine Functionality & Distribution Risk
Determine Overall System Risk
Do we have to
validate?
Determine Document Level
Identify Validation Documents
Identify extent of
test scripts
START
1 2 3 4
5678
Do We Have to Validate?
• Is this a GxP system - GLP, GCP or GMP?– Laboratory, Clinical or Manufacturing
• Does the system affect Patient Safety, Product Quality or Data Integrity?
1
What is the Quality & Regulatory Risk?• Evaluate the criticality of the data processed by the
system as Low, Medium or HIGH risk
• For each risk category identify examples of GMP, GCP and GLP records in the system– HIGH: GMP (Master records, batch records, etc.)
– Medium: GCP (Clinical trial patient data)
– Low: GLP (Training records, facilities records, etc.)
• Use the highest identified risk level
2
What is the Functionality Risk?• Assign values based on functions the system
performs– Electronic data capture 3 – Data entry/modification/calculation 3
– Submission creation 2– Data analysis and reporting 2
– Data/document storage 1– Data browsing 1
3
What is the Distribution Risk?
• Custom system 5• Multi-industry, limited use 4
• Regulated industry, limited use 3 • Regulated industry, broad use 2
• Multi-industry, broad use 1
3
What is the Complexity?
• Add Functionality and Distribution Risk
• Complexity LevelHIGH = 7 to 8
Medium = 4 to 6
Low = 2 to 3
3
Determine Overall System RiskQuality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9
4
Determine Overall System RiskQuality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9
HIGH + Low = HIGH 8Medium + HIGH = Medium 7
4
Determine Overall System RiskQuality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9
HIGH + Low = HIGH 8Medium + HIGH = Medium 7
Medium + Medium = Medium 6Medium + Low = Medium 5Low + HIGH = Medium 4
4
Determine Overall System RiskQuality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9
HIGH + Low = HIGH 8Medium + HIGH = Medium 7
Medium + Medium = Medium 6Medium + Low = Medium 5Low + HIGH = Medium 4
Low + Medium = Low 3Low + Low = Low 2
4
HIGH
Low
MEDIUM
MEDIUM HIGH
Low
MedDRAWHO Drug
SAS
Clin Data Mgt
Elec Data Cap
ERP
Doc Mgt
Submission Publishing
Drug StabilityClin
SuppliesSoftware Upgrade
Clin Trial Mgt
Hardware/ Network Upgrade
Excel
Scanner
Drug Coding
DrugSafety
RISK
COMPLEXITY
HIGH
Low
MEDIUM
MEDIUM HIGH
Low
RISK
COMPLEXITY
MedDRAWHO Drug
SAS
Clin Data Mgt
Elec Data Cap
ERP
Doc Mgt
Submission Publishing
Drug StabilityClin
SuppliesSoftware Upgrade
Clin Trial Mgt
Hardware/ Network Upgrade
Excel
Scanner
Drug Coding
DrugSafety
HIGH
Low
MEDIUM
MEDIUM HIGH
m-m
l-H
m-H
H-HH-mH-l
l-m
m-l
l-l
Low
COMPLEXITY
RISK
MedDRAWHO Drug
SAS
Clin Data Mgt
Elec Data Cap
ERP
Doc Mgt
Submission Publishing
Drug StabilityClin
SuppliesSoftware Upgrade
Clin Trial Mgt
Hardware/ Network Upgrade
Excel
Scanner
Drug Coding
DrugSafety
What do we know at this point?• We know the overall system risk level
–H - H, H - M, H - L, M - H, M - M, …. L – M, L – L
• We have a value for the overall system risk level
Are we there yet?
What Kind of Change is Happening?
• Release (Major) – x.0: Adding functionality;– Change value of 1.0
• Upgrade (Minor) – x.y: Changing functionality;– Change value of 0.8
• Patch – x.y.z: Not changing functionality;– Change value of 0.5
• Hot Fix – x.y.z.w: Smell for smoke;– Change value of 0.2
5
Determine Overall Document Level
System Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2
H - HH - M
H - LM - H
M - MM - LL - H
L - ML - L
6
Determine Overall Document Level
System Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2
H - H 10 10.0 8.0 5.0 2.0H - M 9 9.0 7.2 4.5 1.8
H - L
M - H
M - MM - LL - H
L - ML - L
6
Determine Overall Document LevelSystem Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2
H - H 10 10.0 8.0 5.0 2.0H - M 9 9.0 7.2 4.5 1.8
H - L 8 8.0 6.4 4.0 1.6M - H 7 7.0 5.6 3.5 1.4
M - MM - LL - H
L - ML - L
6
Determine Overall Document LevelSystem Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2
H - H 10 10.0 8.0 5.0 2.0H - M 9 9.0 7.2 4.5 1.8
H - L 8 8.0 6.4 4.0 1.6M - H 7 7.0 5.6 3.5 1.4
M - M 6 6.0 4.8 3.0 1.2M - L 5 5.0 4.0 2.5 1.0L - H 4 4.0 3.2 2.0 0.8
L - M 3 3.0 2.4 1.5 0.6L - L 2 2.0 1.6 1.0 0.4
6
Identify Validation DocumentsDocument \ Value 10 - 9 8 - 7 6 - 4 3 - 0
Validation Plan I I I C OUser Requirements I C C C C O
Risk Analysis I C C C O
Functional Specs I C C O O O
Design specs I C O O O
Trace matrix I C C C O
I = Individual, C = Combined, O = Optional
7
IT’S A GREY AREA!
Identify Validation DocumentsDocument \ Value 10 - 9 8 - 7 6 - 4 3 - 0
IQ Protocol/Scripts I C I C C C OOQ Protocol/Scripts I C I C C C O
PQ Protocol/Scripts I C I C C C O
DM Protocol/Scripts I I C C C O
Validation Summary I I I C O
Production Release Memo
I O I O I O O
I = Individual, C = Combined, O = Optional
7
Now where are we?
• We identified what validation documents are needed for the project
• We have QA buy-in
Validation documents
Identify Extent of Test Scripts
• Determine level of testing• Apply a Functional Risk Assessment to all user
requirements
• Prioritize requirements as HIGH (H), Medium
(M) or Low (L)• Assess requirements as regulatory/ business risk
Critical (C) or Not critical (N)
• High and Critical are the highest risk
8
Prioritize Requirements (Reqmnt)
• HIGH - H (Mandatory): Reqmnt must be present for the system to operate
• Medium - M (Desirable to have): Reqmnt need not be present for the system to operate but it is a ‘nice to have” feature
• Low - L (Low impact/may be useful): Reqmnt need not be present for the system to operate, may be useful and has low impact to the prime user department
• Null - N (Not used): Reqmnt will not be used and will not be validated
8
Prioritize Risk of the Requirement
• Critical (C): If this Reqmnt is not done there is a risk of non-compliance. Reqmnt is critical for business reasons (data are correct, system is Web available, data entry and output are correct, etc.)
• Not critical (N): Reqmnt has no regulatory or business risk associated with it
8
To Test or Not to Test?Priority (H/M/L)
Risk (C/N)
Test (Y/N)
H C YM C YL C Y-- -- --H N YM N NL N NN N N
8
Assessing Changes to a Validated System for Company XYZ
Description Priority (H/M/L)
Risk (C/N)
Test (Y/N)
Changes to the safety database schema and tables
H C Y
Ability to capture partial dates M N NAble to print reports for FDA submissions on a representation of the revised MedWatch form
H C Y
Customer fixes for expedited reports M N NLab test name encoding from Case Form L N NElectronic Submission: Updated feature not used
-- N N
Interactive Exercise
• Our vendor sends us a minor upgrade to our Drug Safety system
• It affects our existing requirements
• What validation documents are needed?• What should be tested?
The Next Step Is Yours
OR
Bonus Information: References• “General Principles of Software Validation, Final Guidance for
Industry and FDA Staff”, FDA, Jan 11, 2002. www.fda.gov• Annex 11 Computerised Systems• Computer Validation: The 100 Worst Mistakes You Can Make,
Tamara Follett, 2003• The Computer System Risk Management and Validation Life Cycle, R.
Timothy Stein, 2006 (many references)• GAMP 5 A Risk-Based Approach to Compliant GxP Computerized
Systems, ISPE, 2008• “GIP Guidance Module 1: Validation and Verification”, HIMSS, 2011,
himss.learn.com/learncenter.asp?id=178409&page=184• Google “Software Compliance Science John Murray FDA” for a PDF
file: www.softwarecpr/.../download.asp
Bonus Information: References• “RAMP: An Approach to Risk-Based Computer System Validation
and Part 11 Compliance”, Drug Information Journal, Vol 41, pp 69-79, 2007 (Use to estimate Quality, Regulatory, Functionality and Distribution Risk)
• “Validation of Software for Regulated Processes”, Assoc. for Advancement of Medical Instrumentation, 13-Dec 2007
• “Effective and Practical Risk Management Options for Computerised System Validation,” R. D. McDowall, Quality Assurance Journal, Vol 9, Issue 3, 2005
• IT Pharma Validation Europe: http://www.ccs-inspired.com• Ask About Validation: http://www.askaboutvalidation.com• Risk Doctor Network: http://www.risk-doctor.com• Compliance Webinars: http://www.complianceonline.com/
Four Assessment Questions• Does the system automate a process or
manage date regulated by these regulations?
– GMP (21CFR 210 and 211)– GCP (21 CFR 312)– GLP (21 CFR 58)– PDMA (21 CFR 203)
• If yes to any of the above, it is GxP
Bonus Information: Validation Life Cycle
User Requirement Specifications
Functional Specifications
IT Computer System Validation and Document Flow: Key Validation Documents
IDENTIFY REQUIREMENTS WRITE PROTOCOL & TEST SCRIPTS
EXECUTE TESTS SUMMARIZE TEST RESULTS WRITE FINAL REPORTS
Optional: C-Risk Analysis, D-Training Plan, E-Detailed Design Specs., F-Configuration Spec.,
G-Training & Manuals, H-Migration Qualification Protocol,
I-Draft WIs/ SOPs, J-Migration Qualification Report
Administrative: A-GxP Assessment, B-IT Statement of Work, K-MS Project Plan
Post Implementation
Folder Organization1 Admin2 Pre xQ3 IQ4 During Xq - Configuration & Training - SOPs and WIs - Migration5 OQ6 PQ7 Post xQ8 Post Impl
2
2
Performance Qualification Test
Execution
6 Performance Qualification
Summary Report
6
IQ Execution(Prod. Env.)
3IQ Summary
Report(Prod Env.)
3
2
Performance Qualification Protocol
& Test Scripts
6
IQ Protocol & Test Scripts
(Production Env.)
3
CHANGE CONTROL
8
** EVALUATE ** ** PLAN **
2
Supplier Audit
7
Design Specifications
Validation/ Qualification
Summary Report
7
Installation Qualification Protocol & Test Scripts
(Validation Env.)
3
Installation Qualification Test
Execution(Val Env.)
3Installation
Qualification Summary Report
(Val Env.)
3
Validation/ Qualification Project
Plan
2
** REQUIREMENTS ** ** EXECUTION **** DEPLOYMENT **
** MAINTENANCE **
** EXECUTION **
- Major release- Added functionality
Training Records
7
Support Model
7
7
Final WIs/ SOPs8
** RETIREMENT **
Operational Qualification Protocol
& Test Scripts
5
Operational Qualification Test
Execution
5 Operational Qualification
Summary Report
5
B
1
K1
C
2 D
2
E4
F4
G4
H
4
I
4
F
4
J
7
Pre
J
7
J
7
Post J
7
IQ
IQ
F4
F
4
I-Draft WIs/ SOPs, J-Data Migration Report
G-Training & Manuals,
F-Configuration Spec., E-Detailed Design Specs.,
H-Data Migration Protocol,
D-Training Plan,
G
4
Requirements Trace Matrix
A
1
Production Release memo with
Workarounds, if any
![Research Paper Risk Stratification of Upper Urinary Tract ... · predictive models was analyzed using the validation set by calculating the overall concordance index [19, 20]. Risk](https://img.pdfslide.net/doc/110x75/5e9736965758eb3bd10575ed/research-paper-risk-stratification-of-upper-urinary-tract-predictive-models.jpg)
