VALIDATIONOF MANUFACTURING EXECUTION … · MANUFACTURING EXECUTION SYSTEMS 6. WHATISANMES •Manufacturing Execution System •Takes in vast quantities of data coming from the PLCs

R A U L S O T O , M S C , C Q EI V T O R L A N D O 2 0 1 7

VALIDATION OF MANUFACTURINGEXECUTION SYSTEMS (MES)

1A S Y S T E M S D E V E L O P M E N T L I F E C Y C L E ( S D L C ) A P P R O A C H

• The contents of this presentation represent the opinion of the speaker; and not necessarily that of his present or past employers.

2IVT MES ORLANDO (C) 2017 RAUL SOTO

ABOUT THE AUTHOR

• 20+ years of experience in the medical devices, pharmaceutical, biotechnology, and consumer electronic industries

• MS Biotechnology, emphasis in Biomedical Engineering• BS Mechanical Engineering• ASQ Certified Quality Engineer

• I have led validation / qualification efforts in many scenarios:• High-speed, high-volume manufacturing and packaging equipment• Enterprise resource planning applications (i.e. SAP)• IT network & infrastructure• Quality Systems Software (CAPA / NCs; pharmacovigilance; eDMS; eLMS; etc.)• Mobile applications• Laboratory : information systems and equipment / computerized analytical instruments• Product improvements, material changes, vendor changes

• Contact information:• Raul Soto [email protected]


4

http://camstar.industrysoftware.automation.siemens.com/user-groups/manufacturing-star-awards/#Johnson-Johnson

IVT MES ORLANDO (C) 2017 RAUL SOTO

OUTLINE

• Introduction• What is an MES

• Benefits

• ANSI / ISA – 95

• eDHR

• Scope: • What should be included in the

Validation project

• SDLC Approach• Description and deliverables

• Validation Deliverables• Planning and Strategy

• Requirements, Design, Trace Matrix

• Testing

• Reports, Procedures, and Governance

• Interfaces

• Going Live• Governance

• Hypercare


INTRODUCTION


6M A N U F A C T U R I N G E X E C U T I O N S Y S T E M S

WHAT IS AN MES

• Manufacturing Execution System• Takes in vast quantities of data coming from the PLCs

and SCADA/HMI systems• Converts that into useful information about production

ops• Scheduling• Materials handling• Quality samples


ERP MES CONTROLS


ANSI / ISA – 95 CONTROL HIERARCHY LEVELS

• Level 0: Physical production process

• Level 1: Sensing/manipulation of production process

• Level 2: Monitoring, supervisory control and automated control of production process (i.e. SCADA HMI systems)

• Level 3: Workflow / recipe control to produce desired products, maintain records and optimize production process. Timeframe: shifts, hours, minutes, seconds (i.e. MES, LIMS)

• Level 4: Establish basic plant schedule – production, material use, delivery, shipping. Timeframe: month, weeks, days, shifts (i.e. ERP)


ANSI / ISA – 95 CONTROL HIERARCHY LEVELS

IVT December (c) 2017 Raul Soto 10

http://w

ww

.we

rum.c

om

/en/m

es/p

rod

ucts/p

as-x/isa

/isa.jsp

BENEFITS OF MES

• Provide complete history of manufactured lots/ batches/ units

• Electronic Device History Record / Batch Record

• Electronic approvals and audit trails

• Full traceability of product to raw materials and manufacturing process parameters at every stage

• Reduce use of paper


BENEFITS OF MES

• MES system is not just a piece of fancy technology

• The BUSINESS VALUE of an MES comes from the integration of data from multiple information sources: production data, quality data, business data, controls data, etc.

• Materials consumed, equipment used, parametric data, exceptions, date / time stamps

• Enforces as-designed process

• Ensures the right procedures, operators, equipment materials are used

• Enforces right sequence of operations

• Can track equipment usage and communicate with eCMMS


EDHR

• eDHR: Electronic Device History Records

• paperless, electronic systems within MES

• enforce production processes

• capture all information associated with as-built production records.

• provides the error-proofing and real-time visibility necessary to produce consistent product quality each and every time.


EDHR

• If product quality issues arise, eDHR can be used to contain suspect product, either in process or in the field, to take action to address the issue.

• The electronic records stored in eDHR yields information that can also help MD&D companies optimize production processes.



SCOPE : WHAT SHOULD BE INCLUDED INTHE VALIDATION PROJECT

16


M A N U F A C T U R I N G E X E C U T I O N S Y S T E M S

SOFTWARE VALIDATION

• validation, software. (NBS) Determination of the correctness of the final program or software produced from a development project with respect to the user needs and requirements. Validation is usually accomplished by verifying each stage of the software development life cycle. See: verification, software.

• Source: FDA Glossary of Computer System Software Development Terminology http://www.fda.gov/iceci/inspections/inspectionguides/ucm074875.htm


SCOPE: SOFTWARE

• MES Application• Workflows

• Databases / Data warehouse • (i.e. Oracle)

• Middleware • (i.e. Archestra)

• Interfaces • (more on this later!)

• ETL tools• SQL scripts

• Reporting tools not part of the application • Reports created with tools like

Tableau, Cognos, BusinessObjects

• Other ancillary / support software


SCOPE: HARDWARE

• Servers (for all environments)

• Sandbox / Development / QA-Test /

Staging / Production

• Scanners

• Printers

• Bar code readers

• RFID systems

• others


SCOPE

• One or multiple sites?

• One or multiple manufacturing suites / lines (within each site)?

• Are you also decommissioning a legacy MES system?

• Data migration and /or archival

• GAMP 5: COTS vs Configuration vs Customization

• Medical Devices : eDHR • 21 CFR 820.184 - DEVICE HISTORY RECORD


SCOPE

• Define your project scope early on, document all changes

• Map your AS-IS business process workflows (pre-MES), vs your TO-BE

• Include in scope any configuration and/or customizations required (i.e. reports, interfaces)

• Recommendation: Minimize custom-coding, maximize the use of configuration alternatives provided by software.


PROJECT PLANNING

• Involve key stakeholders and end users from the start

• Testing, validation, user training and documentation require time and resources. Make sure you factor that into your project scope from the start.


SDLC APPROACH

23


M A N U F A C T U R I N G E X E C U T I O N S Y S T E M S

WHAT IS AN SDLC

• SDLC : Systems Development Life Cycle

• Series of steps / phases that provide a model for development and lifecycle management of an application or piece of software.

• Software validation is NOT a one-shot deal.

• Validated software and its documentation require continuousupdates and improvements for the life of the system

• End-of-life has to be managed• Data migration or archival• Hardware decommissioning


SDLC APPROACH


TYPICAL VALIDATION DELIVERABLES


Validation Plan

Assessments

Requirements

Design

Development

SAT / FAT

IQ

OQ / System Testing

PQ / UAT

Traceability Matrix

Procedures

Validation Report

PURCHASING REQUIREMENTS

• NOT the same as your validation requirements

• Develop high level requirements to be used as part of the purchasing decision

• Vendor assessment should include compliance with these requirements

• Make sure you have a good idea of how much configuration and/or customization is needed to meet your main requirements• MINIMIZE the amount of customization / custom code

• These requirements will also be the basis for FAT / SAT

• SLA : Service Level Agreement


DEVELOPMENT / CONFIGURATION

• Development testing occurs before validation• COTS functionality + configuration

• Vendor or hired consultants

• Unit testing, test configuration, integration testing

• If custom coding is required, development testing should be more comprehensive

• Use bug tracking software (i.e. FogBugz, Redmine, Debbugs) to keep track of bug resolution

• Does not need to be formally documented, but IT HELPS

• Have Coding standards and follow them!


VALIDATION DELIVERABLES

29M A N U F A C T U R I N G E X E C U T I O N S Y S T E M S


TYPICAL LIST OF VALIDATION DELIVERABLES

• Assessments • (Safety, 21CFR Part 11, Risk, Compliance,

Vendor etc.)

• Validation Plan

• Requirements (URS, FRS)

• Design (DS, FDS, TDS, DDS)

• Traceability Matrix

• Testing protocols and scripts• Development *

• Integration *

• FAT / SAT *

• Installation

• OQ / PQ or System Testing / UAT

• Testing Reports

• Training

• Procedures• Operational

• Administration

• Change Control

• Validation Report


* Do not fall strictly under validation

31

ASSESSMENTS

• Vendor Assessment

• Compliance Assessment (i.e. GxP, SOX, etc.)• 21 CFR Part 11 ER/ES (especially on Medical Devices)

• GAMP category

• Information Security

• Safety / EHS

• Risk Analysis / FMEA

• others


VALIDATION PLAN & REPORT


VALIDATION PLAN

• This is where you establish WHAT needs to be validated, and HOW it will be validated.

• Get everyone’s buy-in and commitment to the Validation Plan from the beginning

• Update as needed, following your organization’s formal change control process for documentation

• Define clear expectations for all roles

• Definitions of deliverables, author and approvers for each


VALIDATION PLAN

• System Overview

• Scope of Validation

• Roles and Responsibilities

• Assessments Required

• Validation Deliverables, author and approvers

• Validation and Testing Strategy

• Acceptance Criteria

• How to deal with• Testing defects and deviations

• changes in scope and strategy


VALIDATION REPORT

• Final scope

• List of Actual Deliverables• Document Numbers

• Date of Completion

• any additions or deviations from planned

• Summary of testing results per protocol; any additions or deviations

• Summary of defects and discrepancies, and resolution status

• Statement declaring the validated state of the system

• Release to Production


RESOURCES

• You will need resources from multiple disciplines (i.e. QA, IT, Engineering, Operations, technical writers, etc.)

• Plan which resources you need, when will you need them, and for how long

• This will help you budget for consultants, and borrow internal resources. “I need two validation resources, 25-30% of their time, during August and September” is a different conversation than “I need to borrow some of your people for the MES project”

• PM tools -such as MS Project- help


REQUIREMENTS

• Types:

• User Requirements

• Functional Requirements

• Regulatory Requirements

• Performance Requirements

• Security Requirements

• Classify by type and criticality

• Low criticality : nice to have

• Medium criticality: important but not critical

• High criticality: must have

• All regulatory requirements should be classified as HIGH criticality

38

• WHAT do you want your MES system to do?


REQUIREMENTS

• Clear• Complete• Correct• Consistent• Concise• Prioritized

• Relevant• Feasible• Verifiable• Modifiable• Traceable• Unique

39

Requirements should be


REQUIREMENTS

Written definition of the MES software functions:

• Proposed Workflows and Use Cases

• How users will interact with the system

• Functions that the MES software will perform

• Expected system inputs and outputs

• Reports the system is expected to generate, contents

• Interfaces (user, external)

• Required alarms and checks

• Performance: data throughput, reliability, timing

• Coding standards


REQUIREMENTS

• Required response times

• Intended operating environment (hardware platform, software operating system, middleware, database software, etc.)

• All ranges, limits, defaults, specific values that the MES software requires

• Mirroring & Replication of main and transactional databases

• Backup, restore, disaster recovery

• User Roles that need to be defined in the system

• Any safety-related specifications

• Part 11: Electronic Records & Signatures requirements

• Dates (Manufacturing, Expiration): based on local time or UTC?41IVT MES ORLANDO (C) 2017 RAUL SOTO

REQUIREMENTS: UDI

Unique Device Identification

• US Initiative to assign a unique identifier to all medical devices

• Signed into law on 2007, mandatory since 2013 FDA Final Rule

• Label of a medical device must have a unique identifier

• Unique identifier should ID device through distribution and use

• Unique identifier must include lot / serial number

• Typical elements:• Graphics and symbols

• Regular and 2D barcodes

• Lot / batch / serial number, expiration date, date of manufacture



DESIGN

• HOW does your MES as installed and configured, meet the Requirements?

• Design Specifications describe the actual software solution:

• Out-of-box / configured / custom-coded

• Address all requirements, individually or in groups

• Design documentation may include one or more of the following:

44

Design Specification

Detailed Design Specification

Technical Design Specification

Database Design Specification

Software Design Specification

Hardware Design Specification

Architecture Design

Security Design

InterfacesIVT MES ORLANDO (C) 2017 RAUL SOTO

DESIGN

Design documentation should include, at least:

• Actual workflows and use cases, as configured (or coded)

• System architecture

• Modules

• Screens

• Formulas, algorithms, and logic used

• Data structures, data flow diagrams

• Supporting software that is required for MES operation (i.e. Archestra)

• Hardware required

• Parameters that are measured or recorded


DESIGN

• Definition of control and data variables, where used

• Messages: Errors, alarms, warnings

• User roles, as configured

• Physical security

• Information security

• Actual reports as configured

• Interfaces, as configured


REQUIREMENTS TRACEABILITY MATRIX

• The RTM should enable us to trace:• From individual Requirements to their

specific Design Element(s) where the requirement is addressed

• From Design Elements to the specific Test Script(s) where they are being challenged.

47

We should be able to trace back and forwardRequirement Design elements Testing


REQUIREMENTS TRACEABILITY MATRIX

• Requirements => start RTM draft

• Design documents => update RTM draft

• Test scripts => finalize RTM

• Final RTM should be approved before the system is released for production

• RTM should be updated as part of system change control every time requirements, design, or test scripts are changed, added, removed.


LIVING DOCUMENTS

• The following documents should be treated as “living documents” and maintained up to date throughout the life of the system

• Requirements • Design• Traceability Matrix• Risk Assessments (i.e. FMEA)


TESTING50M A N U FA C T U R I N G E X E C U T I O N S Y S T E M S


TESTING

• Protocols, Test Scripts, Reports

• Pre- and post- approved

• Consistency in roles pre- and post- approving is important• Minimum: System Owner, Business Process Owner, QA

• Avoid conflicts of interest: Approvers cannot execute, testers can’t approve

• Dependencies and order of testing must be clear, evident, obvious

• Enforce Good Documentation Practices (GDP)

• Enforce the use of standard templates

• Keep evidence (printouts, screenshots, labels, etc.)


TESTING ENVIRONMENTS

• Development / Sandbox • For DEV testing• Qualification not required

• Test / Staging• For your formal validation (OQ, System Testing, UAT)• Qualified• Functionally equivalent to your PROD environment

• Production Environment• Live system• Performance testing• Qualified


INSTALLATION QUALIFICATION

• Installation Qualification• Document / verify the correct installation and configuration of

all software and hardware components, as per the Design Specification

• List actual software components and objects installed: name, version, location

• List actual hardware installed: name, model, quantity, S/N, location

• Turn-key test to ensure the system is ready for OQ / System testing


OPERATIONAL QUALIFICATION

Operational Qualification• Ensure system can perform under controlled conditions, non-

saleable product

• Document that system is installed and configured as per Design Specifications

• AND complies with requirements as per Requirements Specifications

• Testing will challenge all requirements, interfaces, reports, etc.

• Can test each subsystem individually

• Positive vs Negative testing

• Formal change control



• System Testing• Test and verify that the system as integrated is functionally

complete

• Challenge compliance with functional and non-functional specifications

• Integration of MES with interfaces

• You can test individual functionalities, then end-end process workflows



Risk-based testing:

1. COTS elements: • positive testing of all workflows, and alternate flows

2. Configured elements: • Positive testing• Negative testing• Boundary testing

3. Custom: • Same as Configured elements, PLUS

• Code reviews against coding standards



• Tests functional requirements, configuration, security, ER/ES, compliance• May include positive testing, negative testing, boundary testing, interface

testing

• Positive Testing:• Ensures that system performs as intended, using normally expected inputs

• Negative Testing:• Ensures that system doesn’t accept invalid inputs

• Boundary Testing:• Challenges that performance is as expected when specific variables are set

to their max / min values.• Interface Testing:

• Tests that system components can pass data correctly to one another


PERFORMANCE QUALIFICATION

Performance Qualification / Production Qualification• Test that system is able to function under normal manufacturing

conditions

• End-to-End challenge of the complete process flow

• Normal mix of lots, products, operators, shifts, etc.

• Performance testing / load testing

• SOPs must be approved and effective

• Can use saleable product if company procedures allow for it



• User Acceptance Testing• Testing focused on user-related requirements

• Challenge that the MES system is capable of supporting your normal business process

• Test cases should address ease of use from the standpoint of operators, QC technicians, etc.



• Tests functional requirements, configuration, security, ER/ES, compliance• May include positive & negative testing, business process testing, end-to-end

testing, stress testing, performance testing

• Business Process Testing:• Verifies that the system works as intended following business process flows

• End – to – End Testing:• Verifies that the system is capable of supporting the intended process flows, from

beginning to end; ensure data integrity and that the correct data passes between components and interfaces.

• Performance Testing:• Verifies system stability, resource usage, and responsiveness under specific workloads.

• Stress Testing:• Tests system’s performance beyond the limits established in the specified requirements


TESTING EXECUTION

• Ensure test sets and scripts are executed in correct order

• GDP (Good Documentation Practices) are IMPORTANT

• Include evidence: screenshots, printouts, labels, etc.

• Changes to approved protocols / test scripts must follow formal documentation management (version up, re-approve, etc)

• Coordinate execution dates / times with owners of systems MES interfaces with

• Follow process for handling testing defects / deviations

• If you execute PQ / UAT with real product, QA must provide disposition of such product


TESTING TOOLS

• There are electronic testing tools that can be used in substitution of paper test scripts• Example: HP Applications Life Management (ALM), Valgenesys

• MUST validate these tools BEFORE you use them

• Pros: reduce GDP errors, standardize testing process, enforce use of correct templates, enforce approvers rules, manage test defects, can search for documents electronically

• Cons: administration, training, cost, less flexibility, maintenance, fixes / upgrades

• Tools become your official repository of validation documentation

• 21 CFR Part 11 full compliance may require additional wrap-around software solution


PROJECT CHANGE MANAGEMENT

• You must have a formal change control process while in the project, not just after go-live.

• Design freeze date – enforce it!• Any design changes requested after this date => after Go Live

• Except for critical (regulatory, business) changes


CHANGE CONTROL

• All changes should be assessed, documented, pre-approved, tested, post-approved

• Assessment

• Nature and scope of the change

• Risk assessment

• Documentation that needs to be updated (Requirements, Design, RTM, FMEA, etc.)

• Testing

• Functional tests

• Regression tests


INTERFACES

65M A N U FA C T U R I N G E X E C U T I O N S Y S T E M S


INTERFACES

• Do NOT underestimate interfaces, especially to LIMS and your ERP.

• ALL interfaces must be part of the validation effort

• Define and document all your interfaces at the Design stage

• What data passes MES External system

• Test your interfaces configuration in Development testing

• Test your interfaces functionality explicitly in OQ / System Testing


SAMPLE INTERFACES DIAGRAM


ERPERPGlobal

Planning

SCADA / HMI

PLCs

Raw Materials Tracking

Distribution Control

Change ControlLIMS

MES

Lot production quantityand status

Audit DataData

Lot Info

POSSIBLE INTERFACES

• ERP (i.e. SAP, JD Edwards)

• Master production schedule

• Lot quantity and status

• Bill of materials

• LIMS (i.e. Labware, NuGenesis)• QC Audit data

• Manufacturing lines, SCADA / HMI (i.e. Wonderware, RSView)• Manufacturing orders

• Bill of materials

• Lot data

• Raw Materials Tracking software• Data exchange of r.m. lots used

• Documentation eDMS(i.e. Documentum)• Change control and procedures data

• Product hold information

• Distribution Control system• QC release data

• Lot data


GOING LIVE

69M A N U FA C T U R I N G E X E C U T I O N S Y S T E M S


GOVERNANCE

• Procedures• System Administration

• Operation

• Change Control

• Maintenance

• Training

• Security and Access Control

• Configuration Management

• Backup / Restore / Disaster Recovery

• others


GOVERNANCE

• “Hypercare”

• SLA with vendors

• Change Control

• Support structure

• Periodic Review (and revalidation)


REFERENCES


https://www.fda.gov/RegulatoryInformation/Guidances/ucm085281.htm

REFERENCES

• http://ispe.org/gamp-good-practice-guide/manufacturing-execution-systems


REFERENCES

• General Principles of Software Validation; Final Guidance for Industry and FDA Staff, Jan 11 2002• http://www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm126955.pdf

• ISPE GAMP5 Guide: Manufacturing Execution Systems• http://ispe.org/gamp-good-practice-guide/manufacturing-execution-systems

• Understanding Manufacturing Execution Systems (MES)• http://www.freedomcorp.com/solutions/qad/White%20Papers/MES%20White%20Paper.pdf

• Unique Device Identification (UDI)• http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/UniqueDeviceIdentification/

• http://www.labelingnews.com/downloads/UDI_eBook.pdf

• Global Unique Device Identification Database (GUDID) Guidance for Industry and FDA Staff, June 27 2014• http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM369

248.pdf


REFERENCES

• US 21 CFR 820 Quality System Regulation• http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=820

• US 21 CFR 11 Electronic Records and Signatures• http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1

• Annex 11 Computerised Systems. Jan 2011 updateEudraLex, Rules Governing Medicinal Products in the European UnionVol. 4 Good Manufacturing Practice, Medicinal Products for Human and Veterinary Use

• http://ec.europa.eu/health/files/eudralex/vol-4/annex11_01-2011_en.pdf

• ISO 13485 Medical devices -- Quality management systems -- Requirements for regulatory purposes• http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=36786

• ANSI/ISA 95 and ANSI/ISA 88 Standards• https://www.isa.org/standards-and-publications/isa-standards/find-isa-standards-in-numerical-order/


QUESTIONS
