Validation SAP

Validation Challenges in FDA-Regulated Industries

George Serafin

Director

Pharmaceutical Industry Business Unit

September 25, 2002

Validation of SAP R/3 in Compliance with FDA 21 CFR Part 11

Institute of Validation Technology

Computer and Software Validation & Electronic Records and Signatures

Dublin, Ireland

SAP AG 2002, IVT ER/ES Conference, Sept. 25, 2002, George Serafin 2

Agenda

Key Considerations1111

3333

2222 Validation Strategy

Validation Best Practices


Source: (Barry Boehm 1981: Software Engineering Economics)

Cost factors to correct a defect:

Why Validation?

REQUIREMENTS 1x

DESIGN 3x

DEVELOPMENT 15x

TEST 45x

POST-RELEASE200x

Costs to Validate

Costs not to

Validate


Form 483 $1000’s

Warning Letter $100,000’s

Product Recall $Millions

Product Approval $Millions

Import Detention $Millions

Consent Decree $150M+

Why The Concern Over FDA Regulatory Compliance?

Costs to Comply

Costs not to

Comply

FDA GMP Non-Compliance Consequences

$500M !!!


FDA Compliance & Validation Trends

Systems-based Inspection Approach

FDA officially adopted systems-based inspection approach in February, 2002.

This inspection strategy requires review of validation and security of computer systems used to support each system (e.g. materials, production, quality, etc.)

Holistic or System-based Approach to Validation

Validation Strategy recognizes the entire SAP system (application and landscape) as a FDA-regulated system.

The industry has adopted this strategy approximately 2 years ago.

This is consistent with the CSV & Part 11 training the FDA investigators are receiving from EduQuest.

Include Part 11 requirements in Validation Strategy

FDA enforcement increasing in frequency and vigilance

FDA investigators being trained on CSV and Part 11

FDA recruitment of 650+ new investigators with an emphasis in technical (computer/engineering) vs science (chem/bio) disciplines


How Does SAP Collaborate With Industry?

• Parenteral Drug Association (PDA) Standing Member of Industry Advisory Board

• Technical Report 32

• Audit Repository Center (ARC) (Audited June, 2001)

• Part 11 Task Force

• International Society of Pharmaceutical Engineers (ISPE)

• GAMP U.S. Chapter

• Institute of Validation Technology

• Presented at April ER/ES Conference

• American Society for Quality (ASQ)

• ADVAMED (formerly HIMA)

• ASUG – America’s SAP User Group

• JSUG - Japanese Pharma User Group

• MSUG - Multinational SAP User Group

• PISUG - Process Industry User Group

• PVG – Pharma Validation Group

• VCI - German Chemical Association

Global, National & Regional User Groups

Industry Groups

MES / EBRMES / EBR

Validation / Part 11

Validation / Part 11

Global Label ManagementGlobal Label Management

Solution Map

Analysis

Solution Map

Analysis

Quality Manage

ment

Quality Manage

ment

TransportationTransportation

Co-ProductsPlanning

and Costing

Co-ProductsPlanning

and Costing

EH&S

EH&S

eCl@sseCl@ss


Modular, Highly Configurable systems, such as SAP, cannot be fully validated by the supplier

(Note: 21 CFR Part 11 Preamble comments 64-68)

Core ProductBusiness Process Business Process Selected

Why Can’t SAP Provide a “Validated” Version?


Suggested SAP Quality Assurance Documentation

HORIZON – Quality Management Manual of SAP Development

DIN EN ISO 9001 Certificate

The Release and Maintenance Strategy of mySAP.com Components - White Paper

Complying with FDA 21 CFR Part 11 Electronic Records; Electronic Signatures; Final Rule – White Paper

http://service.sap.com/

www.auditcenter.com


21 CFR Part 11 Compliance Rollout in 2001

European Pilot - Janssen Cilag (Switzerland)

PISUG (Scottsdale, AZ)

U.S. Pilot - Roche Diagnostics (Branchburg, NJ)

Part 11 Enhancement Workshop - SAP America (NSQ)

Part 11 Workshop for U.S. Customers (NSQ)

SAPPHIRE Lisbon (Lisbon, Portugal)

Part 11 Enhancement Workshop for European Affiliates (Walldorf)

Part 11 Workshop for European Customers (Walldorf)

SAPPHIRE Orlando (Orlando, Florida)

PDA Audit of Part 11 Development (Walldorf)

Part 11 Workshop Hosted by SAP Denmark

“SAP is to be congratulated on producing such a comprehensive solution to meet the requirements of 21 CFR Part 11.” - PVG

Part 11 Assessment Nov 2001

PVG Part 11 Assessment (Walldorf)


21 CFR Part 11 Compliance Rollout in 2001

http://service.sap.com/pharmaceuticals

Part 11 Web Page Accessed By 160+ Users Representing 19 different countries

North AmericaLatin AmericaSouth AmericaEMEAAsia

Whitepaper and Claim of Compliance Widely Accepted by Industry


GMP systems such as SFC and LIMS are implemented, validated, and maintained separately as part of a distributed technical infrastructure with minimal interfaces to other non-GMP systems. Non-GMP systems such as financials are not subjected to any validation requirements.

Validation Scope - Old Paradigm for Stand-alone Systems

Old Paradigm


GMP and non-GMP business processes are integrated within the R/3 platform (ex. GR of material includes MM and FI)

GMP & non-GMP data are maintained within the same R/3 objects and data structures (ex. Material master, process order)

New Paradigm for Integrated Systems

Validation must ensure data integrity and complete business process integration

New Paradigm


SAP Strategic Approach to Manufacturing Execution

Plant Logistics

Process Control

Electronic Batch Record

SAP

Middleware

Partner Independent

Warehouse Management Batch Management

Process Instruction Sheets

SA

P P

ort

al

SAP Standard Interfaces

Weigh System

Master Data

Order Management

Recipe Management

Finite Scheduling (PP-DS)

Equipment Status (Logbook)

Functional ModelFunctional Model

Supply Chain Execution

Document Management

Problem Reporting

Quality Management


Factors That Influence Any Integration Strategy

Redundancy or Duality of Data (Master and Transactional) Change Management Data Maintenance

Complexity of Infrastructure No. of systems and interfaces to be maintained

Regulatory Compliance especially 21 CFR Part 11 Where did the data originate? Where was it communicated? Where is it being maintained?


Industry Trends – Key Trends for SAP

Cost Reduction

Material Flow

Customer RelationshipManagement

Plant Engineering Operations Distribution

Revenue Driving

Research &Development

Intellectual Flow

• Enterprise extends into R&D

• Expansion of R/3 footprint to support Part 11 remediation

• Use of SAP as pilot application for PKI

• Manufacturing Execution / EBR

• 2 Customers Live

• Intensified interest concerning MES/EBR

• APO in Pharma – Several Live many in pilot

• Enterprise Quality Management

• Supplier Quality Management

• Enterprise LIMS

• Problem Reporting / CAPA

• PLM for Enterprise DMS

• Use of BC Set Technology to document configuration

• Growing interest for analytical CRM


SAP R/3 Enterprise

SAP R/3

Enterprise

Core

The name of the next version of SAP R/3 is SAP R/3 Enterprise

Available to all SAP Customers as part of maintenance

Enterprise Extensions Provide an Opportunity to Reduce Validation Impact

SAP R/3 EnterpriseSAP R/3 SAP R/3

EnterpriseEnterpriseExtenstionsExtenstions

SAP R/3 SAP R/3 EnterpriseEnterprise

ExtenstionsExtenstions

SAP R/3 SAP R/3

EnterpriseEnterprise

Core Core

SAP R/3 SAP R/3

EnterpriseEnterprise

Core Core


Agenda


3333




Total System(Computerized System)

7


7

Controlling System(Computer System)

3

Controlled Process6

Hardware2

Software1

Equipment 4

Operating Procedures and Documentation 5

Operational Environment8

... and all the links between the boxes

... and all the links between the boxes

What needs to be validated?


Installation Qualification (IQ)

“Establishing by objective evidence that all key aspects of the process, process equipment, and ancillary system installation adhere to the approved design criteria and that the recommendations of the manufacturer of the equipment have been suitably considered.”

FDA GHTF Study Group 3 - Process Validation Draft Guidance, June 1998.

•Hardware

•Database and application servers (including full system landscape e.g. Dev, QA, Prod.)

•Client workstations

•Network (connectivity, data integrity)

•Peripherals (printers, tape drives, etc.)

•Software

•Operating system

•Application software (DBA, SAP, BW, etc)

•Utility software (backup, archiving, etc.)


Validation Strategy - System Landscape

Development QA/Validation Production

Sandbox

Verification

IDOC

Configuration

Validation

DATA

Training

Backup

Production

Configuration Transported Via TMS

Client Copy


Operational Qualification (OQ)

“Establishing by objective evidence parameters which result in product that meets all pre-determined requirements.”


• Demonstrates that the configured system performs its intended functions as defined by the functional requirements

This includes manual operations that are formalized through SOPs which are established to support the computerized process


Performance Qualification (PQ)

“Establishing by objective evidence that the process, under anticipated conditions, including worst case conditions, consistently produces a product which meets all pre-determined requirements..”


Performance to measure response time and throughput rates under a variety of conditions.

Volume to subject the system to heavy volumes of data similar to the loads expected to be encountered in the course of normal operations.

Stress to examine the behavior of the system at and beyond the limits of its resources by subjecting it to excessively high loads.


ASAP System Development Lifecycle Roadmap

ASAP Roadmap

Project Prep Blueprinting Realization Final Prep. Go-Live

Concept Requirements Design MaintenanceInstallationSystem

TestUnit Test

Build

Industry-recognized SDLC


ASAP Project Progress

Va

lid

ati

on

Pro

jec

t P

rog

res

s

Project Preparation Business Blueprint Realization Final Preparation GoLive/Support

System Design Specification

IQ Development

Functional Requirements Specification

OQ Production

Validation Master Plan

IQ QA/Validation

OQ QA/Validation

Generate SOPs

IQ Production

Validation Determination

Validation Summary Report

Verify Production Cutover

PQ Production

Project Milestones

Vendor Assessment

Traceability Matrix

Change Control


System Development Lifecycle - Enhanced “V” Model

DEVELOPMENT

FUNCTIONAL REQUIREMENTSSPECIFICATION

BUSINESS PROCESS MODELSOPS

SYSTEM ACCEPTANCE

TESTING

SYSTEM DESIGN SPECIFICATIONPROGRAM SPECIFICATION

UNIT TESTING

PROGRAM SPECIFICATIONPROGRAMMING STANDARDS

SOURCE CODE

STRUCTURAL CODE REVIEW

CONFIGURE / CODE

SYSTEM PERFORMS INTENDED INTEGRATED FUNCTIONS

APPLICATION SELECTIONUSER REQUIREMENTS

ONGOING LIVE MONITORING & CHANGE CONTROL

CONCEPT &REQUIREMENTS

DEVELOPMENT

OPERATION &MAINTENANCE

DESIGN

TESTING

CODECOMPLIES

TO STANDARDS

VMP

AuditReport

ASAP

ScenarioProcess

1 per Process BPP

BPMLBusiness Process Master List

R/3 IMG

R/3 Test

Workbench

BPP

BPP

BAPI

ABAP

SOP

SOP

SYSTEM MEETS ORIGINAL SELECTIONCRITERIA AND USER REQUIREMENTS

PROCESSES & PROGRAMS PERFORM INTENDED

FUNCTIONS

Best Practices

CATT

BC Sets

R/3 Cross System Viewer


Validation Strategy - 21 CFR Part 11 Compliance

Define 21 CFR Part 11 Requirements in:•Validation Master Plan (VMP)•Functional Requirements Specification•Operational Qualification (OQ)

Conduct GxP Assessment Determine GxP-relevant business processes and R/3 ObjectsGxP Relevance Can Be Determined At:

•Transaction or Object level • Field level

System Testing•Create test objectives to demonstrate 21 CFR Part 11 compliance for each

relevant clause of the regulation

• For example, §11.10(e) §11.10(e) challenge the creation of a secure, computer-generated

time-stamped, audit trail. Audit trail shall contain the user-id, date, time and

the changes that were made. All changes do not obscure or overwrite the

previous information maintained.


GMP Determination Rules – Examples

GMP Determination

RuleQuestion

21 CFR Part 211, Current Good Manufacturing Practice for Finished Pharmaceuticals

R5

Does the process impact the control of components, labeling or packaging materials, including goods receipt, identification, quarantine, storage, handling, sampling, testing, or QA release?

Subpart E – Control of Components and Drug Product Containers and Closures

211.80 General requirements.

211.82 Receipt and storage of untested components, drug product containers, and closures.

211.84 Testing and approval or rejection of components, drug product containers, and closures.

211.87 Retesting of approved components, drug product containers, and closures.

Subpart G – Packaging and Labeling Control

211.122 Materials examination and usage criteria.

211.125 Labeling issuance.

Subpart H – Holding and Distribution

Warehousing procedures.

Subpart J – Records and Reports

211.180 General requirements.

211.184 Component, drug product container, closure, and labeling records.


SAP / FDA CGMP Functionality Matrix For Finished Pharmaceuticals


SAP / FDA CGMP Functionality Matrix For Finished Pharmaceuticals


SAP / FDA CGMP Functionality Matrix For Medical Devices






Validation Strategy - Security

Develop Security Profiles In Accordance With System Development Lifecycle (SDLC)

•Establish Functional Requirements Specification For Job Roles

•Use Business Process Master List (BPML) as the only source of

authorization profile development. This will ensure that unused, non-

validated business processes within SAP R/3, are effectively blocked from

unauthorized access

•System testing of profiles should include negative testing of business

critical transactions (ex. CGMP)

•Profiles should be managed similar to configuration in regards to Change

Control and R/3 Transport Management

Ideally, Users Should Be Trained For All Transactions Within Their Profile(s) §11.10 (i) §11.10 (i)

§11.10 (i) Determination that persons whodevelop, maintain, or use electronicrecord/electronic signature systemshave the education, training, andexperience to perform their assignedtasks.

§11.10 (i) Determination that persons whodevelop, maintain, or use electronicrecord/electronic signature systemshave the education, training, andexperience to perform their assignedtasks.



System Development Lifecycle – Key Deliverables


TestBuild

Deliverable Project Charter

Project Plan

Vendor Qualification

Responsibilities Matrix




TestBuild

DeliverableValidation Master Plan

State intention to validate in accordance with §11.10(a)

Describe use (if any) of electronic signatures

Training Program

Not only CGMP requirement but also §11.10(i)

Functional Requirements Specifications “Business Blueprint”

GxP requirements

Detailed Part 11 requirements Job Roles / User Authorization

Full SDLC for Security

Custom programs and reports Data conversion Interfaces





TestBuild

DeliverableSystem Design Specifications

Business Process Design Custom programs and reports

Programming standards, naming conventions

ER/ES requirements

Data conversion Interfaces Job Roles / User Authorization

Use business process (e.g. transaction) master list as only source of

authorization development

Traceability Matrix

Data integrity

Prevention against alteration, erasure, data loss





TestBuild

Operational Qualification

Describe how ER/ES requirements will be challenged during

testing Unit Test

Create test objectives to demonstrate 21 CFR Part 11

compliance for each relevant clause of the regulation

For example, §11.10(e) §11.10(e) challenge the creation of a secure,computer-

generated time-stamped, audit trail.

Job Roles / User Authorization

Perform manual review (static analysis) of authorization profiles against functional requirements for unit testing.

Deliverable





TestBuild

Manage integration testing in two (2) phases: Application-centric

Focus on how business processes within ERP application

interact with each other

Enterprise-wide Focus on integration of outside systems through manual and

electronic interfaces

Job Roles / User Authorization

System testing of profiles should include negative testing of

business critical transactions (ex. CGMP)

Cutover Strategy and Plan

Deliverable





TestBuild

Various Summary Reports (testing, data conversion, etc)

Validation Report

Includes completed, documented resolution of all deviations from Validation Plan.

Traceability Matrix

Client Operations Manual

Training Materials

Performance Qualification of Production Environment

Post Implementation Review Protocol*

Deliverable





TestBuild

Rigorous Change Control Program Established, streamlined procedure Clear definition and classification of triggers Complete, accurate impact assessments Documented assessment of software fixes from vendor Effective revalidation /regression testing Complete, documentation package of implemented change

Problem Reporting Established, streamlined procedure 2 levels (help desk, System Investigation Reports)

Trend Analysis Periodic review of changes, Help Desk, and incident logs for trends

Deliverable




7


7


3

Controlled Process6

Hardware2

Software1

Equipment 4



What Needs To Be Re-validated?

Releases/versions

Operating systems Database SAP Desktop Applications

SAP fixes Kernel patches Hot packages OSS notes

IMG configuration

Enhancements/Reports

Interfaces



7


7


3

Controlled Process6

Hardware2

Software1

Equipment 4



What Needs To Be Re-validated?

Database & Apps. Servers

Workstations

Network

Peripherals


Agenda


3333




Validation Standards

Version Control

Handwritten Approval

Document Naming/Numbering Convention

Format / Templates

Documentation

Pre-approval Before Execution

Specific Test Instructions

Independence of Review***

Expected Results

Actual Results

“Objective” Evidence of Test Results

Reference to Support Documentation

Test Problem Report / Resolution

Post Approval

Testing

*** Recent FDA Guidance Documents

Guidance for Industry 21 CFR Part 11; Electronic Records; Electronic Signatures Validation

General Principles of Software Validation; Final Guidance for Industry and FDA Staff


Good Documentation Practices

All records should be written in black or blue ink

Any mistake should be crossed-out with a single line, initialed, and dated

First and last names should be printed on records to prevent confusion of individuals with the same initials or names. All records should be initialed/signed and dated.

Standardize on one date format to avoid confusion

No field should be left blank; thus raising the question of whether or not the blank field is a record keeping error. Rather, record N/A for “not applicable” or NR for “not required” where appropriate.

Prohibit use of shortcuts such as arrows drawn through columns or rows or dittos” “ are unacceptable for initials, signatures, and dates.

Draw a diagonal line through remaining unused rows for records and write “N/A”. This establishes confidence that the record is complete and intact. Without the diagonal line, the record can appear ”open” and further information potentially could be added at a later date.


Good Testing Practices

Define a “unit” for unit testing to be a logical grouping of transactions (COR1, COR2, COR6).

Expected Results and Actual Test Results should be quantitative with specific values (rather than pass/fail) wherever possible.

Include both Global Test Data (for entire script) and Test Data required to execute a specific section.

Standardize how the author of the test script is to be documented (last name, 1st initial, last name, full name, initials, etc.)

Ensure that all test scripts have been completely signed and approved prior to execution.

Include traceability matrix to identify business requirements

Record client in which test was executed (this ensures that the specified validation client was used)

Do not leave various rows, columns, and fields blank – this can lead to confusion as to whether or not information should have been recorded.

A copy of the Test Incident Documentation should be included with test report (This will eliminate the frantic search for documents during an inspection)


Good Testing Practices – Continued

The following information is recommended when documenting screen captures for validation testing documentation:

Test script number

Reference to test data set number or expected results number (answers the question “What is the screen shot for?”)

Tester’s initials and date

Page x of y (ensures complete set of screen shots are included)

Circle actual results that satisfy expected output requirements including data, warning or error messages, completion messages at the bottom of the left hand corner of the screen, etc



Validation Is The Responsibility Of The Business Owner(s)

Validate To Satisfy Your Business Requirements - Not FDA

Document What You Do / Do What You Document

There Can Be Only One Project Plan

Include QA/Validation At The Beginning Of Project Preparation And Planning

Communicate The Validation Requirements & Expectations At The Beginning Of The Project

Maintain Only One Set Of Project Standards & Templates

Document Justification For non-GxP Determination

Challenge Configuration - Not Standard SAP

Rigorous Change Control And Problem Reporting

Aggressively Manage Feedback Loop


Validation - Post Implementation

System Must Remain In A “Validated State”

Visibility Of Changes In Development

Change Control System Must Assess All Changes For “Triggers” Which Impact The Validated State Of The System

Re-validation Activities Are Defined By The Impact Assessment And Regression Analysis Conducted During Evaluation Of Proposed Change

Establish An Audit Schedule (begin quarterly then expand to annually) To Review Implemented Changes And Incident Trends

Each Change Documentation Package Should Describe The Whole Change Process And Provide Proof (e.g. documentation) That All Activities Were Completed


Common Weaknesses Observed During Customer Audits

Part 11 Requirements Not Included in the Validation

Requirement Documents Not Being Updated and Maintained

Traceability Matrix and the Ability to Sufficiently Prove Traceability

Approved, Current Validation Responsibility Matrix

Code Review Procedure and Forms (Including a Dead Code Review)

Approved Job Role Requirements for Security Profiles

Adequate Testing of Security Authorization Profiles

Rigorous Change Control Including Thorough Impact Assessments and Documented Complete Closure of Changes

Documented Training Program (Including Level 1 GMP Training) and Training Logs for Project Team Members (This is Also a Part 11 Requirement (11.10i) Hence, More Attention)

Qualifications for ConsultantsThe above information is based upon the personal experience of the presenter both from within industry and SAP. Its purpose is to highlight that most of these same items were emphasized at the training course “21 CFR Part 11 and Computer System Validation Training provided to FDA investigators” conducted by EduQuest, Inc.

Thank You For Your Attention

Questions?

[email protected]

+1 (973) 316-3668