Using Risk Management for Validation

How Much Validation is Enough?

The Breadth and the Depth

Robert Sturmrmsturm@hotmail.com

Estimating the Dilbert Way (1 of 2)

Estimating the Dilbert Way (2 of 2)

Agenda

• Brief Overview of Validation and Verification• Do we have to Validate the System?• Identify Overall System Risk

• Identify Type of System Change• Identify Needed Validation Documents• Identify Extent of Test Scripts

• Workshop

DISCLAIMER

• Everything in this presentation represents my opinion and not that of my employer.

• It’s all about RISK!

Validation and Verification

• Verify each part: Test and document results

• Validation = ∑ verified parts

IQ OQ PQ

TMURS FS

QPP QSR

VERIFY

It’s the entire process!

Potential Validation Documents(GAMP 5)

Validation Plan

User Requirements

Functional Specs

Risk Analysis

Design specs

Trace Matrix

IQ Protocol/ Scripts

OQ Protocol/ Scripts

PQ Protocol/ Scripts

Validation Summary

Data Migration (DM)

Key Validation Deliverables• Number of validation documents

– How many?

• Extent of the test scripts– What and how much to test?

vs.

vs.

How Much? FDA: : Use Risk

How Much? QA vs. IT

QA IT & CUSTOMER

VALIDATION

Validation Estimating Workflow

Determine Type of System Change

Determine Quality & Regulatory Risk

Determine Functionality & Distribution Risk

Determine Overall System Risk

Do we have to

validate?

Determine Document Level

Identify Validation Documents

Identify extent of

test scripts

START

1 2 3 4

5678

Do We Have to Validate?

• Is this a GxP system - GLP, GCP or GMP?– Laboratory, Clinical or Manufacturing

• Does the system affect Patient Safety, Product Quality or Data Integrity?

1

What is the Quality & Regulatory Risk?• Evaluate the criticality of the data processed by the

system as Low, Medium or HIGH risk

• For each risk category identify examples of GMP, GCP and GLP records in the system– HIGH: GMP (Master records, batch records, etc.)

– Medium: GCP (Clinical trial patient data)

– Low: GLP (Training records, facilities records, etc.)

• Use the highest identified risk level

2

What is the Functionality Risk?• Assign values based on functions the system

performs– Electronic data capture 3 – Data entry/modification/calculation 3

– Submission creation 2– Data analysis and reporting 2

– Data/document storage 1– Data browsing 1

3

What is the Distribution Risk?

• Custom system 5• Multi-industry, limited use 4

• Regulated industry, limited use 3 • Regulated industry, broad use 2

• Multi-industry, broad use 1

3

What is the Complexity?

• Add Functionality and Distribution Risk

• Complexity LevelHIGH = 7 to 8

Medium = 4 to 6

Low = 2 to 3

3

Determine Overall System RiskQuality/Regulatory Complexity System Value

HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9

4

Determine Overall System RiskQuality/Regulatory Complexity System Value

HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9

HIGH + Low = HIGH 8Medium + HIGH = Medium 7

4

Determine Overall System RiskQuality/Regulatory Complexity System Value

HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9

HIGH + Low = HIGH 8Medium + HIGH = Medium 7

Medium + Medium = Medium 6Medium + Low = Medium 5Low + HIGH = Medium 4

4

Determine Overall System RiskQuality/Regulatory Complexity System Value

HIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9

HIGH + Low = HIGH 8Medium + HIGH = Medium 7

Medium + Medium = Medium 6Medium + Low = Medium 5Low + HIGH = Medium 4

Low + Medium = Low 3Low + Low = Low 2

4

HIGH

Low

MEDIUM

MEDIUM HIGH

Low

MedDRAWHO Drug

SAS

Clin Data Mgt

Elec Data Cap

ERP

Doc Mgt

Submission Publishing

Drug StabilityClin

SuppliesSoftware Upgrade

Clin Trial Mgt

Hardware/ Network Upgrade

Excel

Scanner

Drug Coding

DrugSafety

RISK

COMPLEXITY

HIGH

Low

MEDIUM

MEDIUM HIGH

Low

RISK

COMPLEXITY

MedDRAWHO Drug

SAS

Clin Data Mgt

Elec Data Cap

ERP

Doc Mgt

Submission Publishing

Drug StabilityClin

SuppliesSoftware Upgrade

Clin Trial Mgt

Hardware/ Network Upgrade

Excel

Scanner

Drug Coding

DrugSafety

HIGH

Low

MEDIUM

MEDIUM HIGH

m-m

l-H

m-H

H-HH-mH-l

l-m

m-l

l-l

Low

COMPLEXITY

RISK

MedDRAWHO Drug

SAS

Clin Data Mgt

Elec Data Cap

ERP

Doc Mgt

Submission Publishing

Drug StabilityClin

SuppliesSoftware Upgrade

Clin Trial Mgt

Hardware/ Network Upgrade

Excel

Scanner

Drug Coding

DrugSafety

What do we know at this point?• We know the overall system risk level

–H - H, H - M, H - L, M - H, M - M, …. L – M, L – L

• We have a value for the overall system risk level

Are we there yet?

What Kind of Change is Happening?

• Release (Major) – x.0: Adding functionality;– Change value of 1.0

• Upgrade (Minor) – x.y: Changing functionality;– Change value of 0.8

• Patch – x.y.z: Not changing functionality;– Change value of 0.5

• Hot Fix – x.y.z.w: Smell for smoke;– Change value of 0.2

5

Determine Overall Document Level

System Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2

H - HH - M

H - LM - H

M - MM - LL - H

L - ML - L

6

Determine Overall Document Level

System Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2

H - H 10 10.0 8.0 5.0 2.0H - M 9 9.0 7.2 4.5 1.8

H - L

M - H

M - MM - LL - H

L - ML - L

6

Determine Overall Document LevelSystem Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2

H - H 10 10.0 8.0 5.0 2.0H - M 9 9.0 7.2 4.5 1.8

H - L 8 8.0 6.4 4.0 1.6M - H 7 7.0 5.6 3.5 1.4

M - MM - LL - H

L - ML - L

6

Determine Overall Document LevelSystem Risk Value Release Upgrade Patch Hot FixChange value: 1.0 0.8 0.5 0.2

H - H 10 10.0 8.0 5.0 2.0H - M 9 9.0 7.2 4.5 1.8

H - L 8 8.0 6.4 4.0 1.6M - H 7 7.0 5.6 3.5 1.4

M - M 6 6.0 4.8 3.0 1.2M - L 5 5.0 4.0 2.5 1.0L - H 4 4.0 3.2 2.0 0.8

L - M 3 3.0 2.4 1.5 0.6L - L 2 2.0 1.6 1.0 0.4

6

Identify Validation DocumentsDocument \ Value 10 - 9 8 - 7 6 - 4 3 - 0

Validation Plan I I I C OUser Requirements I C C C C O

Risk Analysis I C C C O

Functional Specs I C C O O O

Design specs I C O O O

Trace matrix I C C C O

I = Individual, C = Combined, O = Optional

7

IT’S A GREY AREA!

Identify Validation DocumentsDocument \ Value 10 - 9 8 - 7 6 - 4 3 - 0

IQ Protocol/Scripts I C I C C C OOQ Protocol/Scripts I C I C C C O

PQ Protocol/Scripts I C I C C C O

DM Protocol/Scripts I I C C C O

Validation Summary I I I C O

Production Release Memo

I O I O I O O

I = Individual, C = Combined, O = Optional

7

Now where are we?

• We identified what validation documents are needed for the project

• We have QA buy-in

Validation documents

Identify Extent of Test Scripts

• Determine level of testing• Apply a Functional Risk Assessment to all user

requirements

• Prioritize requirements as HIGH (H), Medium

(M) or Low (L)• Assess requirements as regulatory/ business risk

Critical (C) or Not critical (N)

• High and Critical are the highest risk

8

Prioritize Requirements (Reqmnt)

• HIGH - H (Mandatory): Reqmnt must be present for the system to operate

• Medium - M (Desirable to have): Reqmnt need not be present for the system to operate but it is a ‘nice to have” feature

• Low - L (Low impact/may be useful): Reqmnt need not be present for the system to operate, may be useful and has low impact to the prime user department

• Null - N (Not used): Reqmnt will not be used and will not be validated

8

Prioritize Risk of the Requirement

• Critical (C): If this Reqmnt is not done there is a risk of non-compliance. Reqmnt is critical for business reasons (data are correct, system is Web available, data entry and output are correct, etc.)

• Not critical (N): Reqmnt has no regulatory or business risk associated with it

8

To Test or Not to Test?Priority (H/M/L)

Risk (C/N)

Test (Y/N)

H C YM C YL C Y-- -- --H N YM N NL N NN N N

8

Assessing Changes to a Validated System for Company XYZ

Description Priority (H/M/L)

Risk (C/N)

Test (Y/N)

Changes to the safety database schema and tables

H C Y

Ability to capture partial dates M N NAble to print reports for FDA submissions on a representation of the revised MedWatch form

H C Y

Customer fixes for expedited reports M N NLab test name encoding from Case Form L N NElectronic Submission: Updated feature not used

-- N N

Interactive Exercise

• Our vendor sends us a minor upgrade to our Drug Safety system

• It affects our existing requirements

• What validation documents are needed?• What should be tested?

The Next Step Is Yours

OR

Bonus Information: References• “General Principles of Software Validation, Final Guidance for

Industry and FDA Staff”, FDA, Jan 11, 2002. www.fda.gov• Annex 11 Computerised Systems• Computer Validation: The 100 Worst Mistakes You Can Make,

Tamara Follett, 2003• The Computer System Risk Management and Validation Life Cycle, R.

Timothy Stein, 2006 (many references)• GAMP 5 A Risk-Based Approach to Compliant GxP Computerized

Systems, ISPE, 2008• “GIP Guidance Module 1: Validation and Verification”, HIMSS, 2011,

himss.learn.com/learncenter.asp?id=178409&page=184• Google “Software Compliance Science John Murray FDA” for a PDF

file: www.softwarecpr/.../download.asp

Bonus Information: References• “RAMP: An Approach to Risk-Based Computer System Validation

and Part 11 Compliance”, Drug Information Journal, Vol 41, pp 69-79, 2007 (Use to estimate Quality, Regulatory, Functionality and Distribution Risk)

• “Validation of Software for Regulated Processes”, Assoc. for Advancement of Medical Instrumentation, 13-Dec 2007

• “Effective and Practical Risk Management Options for Computerised System Validation,” R. D. McDowall, Quality Assurance Journal, Vol 9, Issue 3, 2005

• IT Pharma Validation Europe: http://www.ccs-inspired.com• Ask About Validation: http://www.askaboutvalidation.com• Risk Doctor Network: http://www.risk-doctor.com• Compliance Webinars: http://www.complianceonline.com/

Four Assessment Questions• Does the system automate a process or

manage date regulated by these regulations?

– GMP (21CFR 210 and 211)– GCP (21 CFR 312)– GLP (21 CFR 58)– PDMA (21 CFR 203)

• If yes to any of the above, it is GxP

Bonus Information: Validation Life Cycle

User Requirement Specifications

Functional Specifications

IT Computer System Validation and Document Flow: Key Validation Documents

IDENTIFY REQUIREMENTS WRITE PROTOCOL & TEST SCRIPTS

EXECUTE TESTS SUMMARIZE TEST RESULTS WRITE FINAL REPORTS

Optional: C-Risk Analysis, D-Training Plan, E-Detailed Design Specs., F-Configuration Spec.,

G-Training & Manuals, H-Migration Qualification Protocol,

I-Draft WIs/ SOPs, J-Migration Qualification Report

Administrative: A-GxP Assessment, B-IT Statement of Work, K-MS Project Plan

Post Implementation

Folder Organization1 Admin2 Pre xQ3 IQ4 During Xq - Configuration & Training - SOPs and WIs - Migration5 OQ6 PQ7 Post xQ8 Post Impl

2

2

Performance Qualification Test

Execution

6 Performance Qualification

Summary Report

6

IQ Execution(Prod. Env.)

3IQ Summary

Report(Prod Env.)

3

2

Performance Qualification Protocol

& Test Scripts

6

IQ Protocol & Test Scripts

(Production Env.)

3

CHANGE CONTROL

8

** EVALUATE ** ** PLAN **

2

Supplier Audit

7

Design Specifications

Validation/ Qualification

Summary Report

7

Installation Qualification Protocol & Test Scripts

(Validation Env.)

3

Installation Qualification Test

Execution(Val Env.)

3Installation

Qualification Summary Report

(Val Env.)

3

Validation/ Qualification Project

Plan

2

** REQUIREMENTS ** ** EXECUTION **** DEPLOYMENT **

** MAINTENANCE **

** EXECUTION **

- Major release- Added functionality

Training Records

7

Support Model

7

7

Final WIs/ SOPs8

** RETIREMENT **

Operational Qualification Protocol

& Test Scripts

5

Operational Qualification Test

Execution

5 Operational Qualification

Summary Report

5

B

1

K1

C

2 D

2

E4

F4

G4

H

4

I

4

F

4

J

7

Pre

J

7

J

7

Post J

7

IQ

IQ

F4

F

4

I-Draft WIs/ SOPs, J-Data Migration Report

G-Training & Manuals,

F-Configuration Spec., E-Detailed Design Specs.,

H-Data Migration Protocol,

D-Training Plan,

G

4

Requirements Trace Matrix

A

1

Production Release memo with

Workarounds, if any