Upload
christopher-newton
View
216
Download
0
Embed Size (px)
Citation preview
Don’t Just “Test”… Validate!!
Agenda
Overview of Testing versus Validation Common Activities and Responsibilities Justification for Validation Discussion
Traditional View of Testing
Project Life Cycle
Design and Coding
Tes
tin
g
Pro
du
cti
on
En
vir
on
men
t
InstallSet For
Year End
Project Life Cycle
Hiring Freeze
Time & Resource Constraints
Design and Coding
Pro
du
cti
on
En
vir
on
men
t
Changes to
RequirementsT
esti
ng Install
Set ForYear End
Risk Factors Examples of Project Risk Factors
SEI identifies at least 10 project risk factors:
1. Over-ambitious Schedule
2. Under-ambitious Budget
3. Over-ambitious or unrealistic expectations
4. Undefined or misunderstood obligations
5. Inadequate software sizing estimate
6. Unsuitable or lack of development process model
7. Continuous requirements change
8. Unsuitable organizational structure
9. Inadequate software development plan
10.Lack of political support
Assess these factors in relationship to the testing effort
Factors Affecting Testing
Short Discussion:
What have you seen?
Testing Objectives
Maximiz
e System Quality
Minimize Testing Effort
Did we build the system right?
What Is Validation?
Providing documented evidence with high degree of assurance the system will consistently produce a product or perform a process which meets the predetermined specifications and quality attributes
Validation
Risk to Business
Business
Needs
Factors Driving Validation
Critical Success
Factors
Relationship of Risk to ValidationD
egre
e o
f V
alid
atio
n
Degree of Risk
Risk tells you•What to address first•What to address most thoroughly
Validation Objectives
Maximiz
e System Quality
Meet Business Needs
Did we build the right system?
Goals of Validation
Control of system development:—Requirements
meaningful, specified, and approved
—Effective process capability of developer
—Existence of and conformance to SOPs
—Effective, appropriate, documented testing
—Configuration management & change control
Control of system use: —Current accuracy and
reliability
—Continued accuracy and reliability
—Management awareness and control
—Auditability
—Data integrity
—Reviewer independence and credentials
What Makes Up A System?
The Computer System
Hardware
Peripherals
Software
Documentation
The Business Environment
Trained Personnel
Procedures Infrastructure
Testing (traditionally) Encompasses…
Documentation
Trained Personnel
Procedures Infrastructure
The Computer System
HardwareSoftware
Validation Encompasses…
The Computer System
Hardware
Peripherals
Software
Documentation
The Business Environment
Trained Personnel
Procedures Infrastructure
CMM Quality Assurance ProgramProgram Owner Accountabilities
QA Accountabilities occur at every touch point
Idea Generation
GeneralRelease
CustomerRequirement
MarketingDesign
Project Orientation
Initiation
Planning
Executing
Closure
Transition
Implementation Operations Support Retirement
Product Process
Pro
ject
Man
agem
ent
Pro
cess
Test Strategy Test Construction
Test Scope
Test PlanningTest Execution
Test Evaluation
QC Process
Construction
Elaboration
Inception
TransitionSDLC
QA & QC Coming Together
Project Orientation Initiation Planning Executing Closure Transition
Quality AssuranceProcess Consultation
Continuous Improvement
Process Audits Defect Analysis Measurement Analysis
Standards Development
Test & Quality Control
Test Planning
Risk Management Defect Management
Dynamic TestingStatic Testing Test Automation
Measurement & Analysis
Forecasti
ngDefe
ct
Resolutio
n
Metrics Test
Executio
n
Metrics Projec
t Planning
Support Defect
Cost
Analysis
Bench
marking
Common Activities
Each Level is Composed of These Activities
Strategy
Design
Execute
Evaluate
Strategy Development
Determine Validation Strategy Perform Risk Assessment Identify Critical Success Factors Set Validation Objectives Define Validation Activities Estimate Time & Resources Document Validation Strategy
Strategy
Design
Execute
Evaluate
Identify Critical Success Factors
Factor
Correctness
Authorization
Data Integrity
Service Levels
Access Control
Methodology
Reliability
Ease of use
Maintainability
Portability
Performance
Extent to which ...
… system must satisfy stated requirements
… processing requires management authorization
… data stored by the system must be accurate
… schedules must be met
… access must be restricted
… development plan must be followed
… system cannot fail during operation
… effort is required to learn and use system
… effort is required to fix errors in the system
… system can operate in multiple environments
… functions must perform within a specified time
Design
Develop/obtain tools for conducting Validation Activities—Deliverables & Phase Checklists—Procedures describing reviews &
inspections—Validation Matrices—Test Cases—Automated Testing Tools—Audit plans
Design
Strategy
Execute
Evaluate
Examples of Validation Objectives
#
1
2
4
3
Validation Objective
Customer orders processed correctly
System is ready to use
Hold is placed on inventory when QC test fails
Increased through-put
Priority
High
High
Med
Low
Completion Criteria
Enter customer order information as described in the validation test cases and verify the results are correct
No open defects of type SEVERE or HIGHPerformance criteria are metUsers are trained, documentation is ready
Enter FAIL test results for productVerify MRP II indicates status as HOLDVerify product is segregated in warehouse
Measure increase by time study
5 Labels printed as req’d LowPrint labels, verify against approved label copy specifications
Examples of Validation Activities
Activity
Requirements Reviews
Design Reviews
Code Walk-throughs
Code Inspections
Performed by
Business Analysts,
Developers,Users
Developers
Developers
Developers
Description
Study & discuss system requirements to ensure they meet business needs
Study & discuss system design to ensure it will support requirements
Informal analysis of code to find defects
Formal analysis of code to find defects, problem areas, coding techniques, design flaws
Deliverable
Reviewed statement of requirements, ready to be translated into design
Design, ready to be translated into software, hardware, documents, etc.
Software ready to be inspected or tested
Software ready to be tested
Examples of Validation Activities
Activity
Unit Testing
Integration Testing
Acceptance Testing
System Testing
Performed by
Developers
Independent Test Team
Users
Description
Test single units of code. Validates that the software performs as designed.
Test related units, programs, modules. Validates that multiple parts of the system interact according to design.
Conducted at end of Integration Testing. Test entire system. Validates system requirements.
Test system to make sure it works in the environment. Validates business needs.
Deliverable
Software unit ready for testing with other components
Portions of the system ready for testing with other portions
Tested system, based on specification
Tested system, based on user needs
RequirementTest Case
1 2 3 4 5 6Validate sign-on
Validate password
Invalid term ident
Password/ Signon error
Validate Main Menu
Re-enter O, X, M, R
Validate Customer #
x
x
x
x
x
x
x
Customer # Required x
Field entry too long x
Customer ___ not found x
Detailed Validation Matrices
Execution
Organize Validation Tools Train Team Members Execute Validation Plan Track Validation Progress Perform Regression Testing Document Results
Execute
Strategy
Design
Evaluate
Evaluation
Evaluate Compliance of System Development
Evaluate Defects Develop Findings and Recommendations Formalize Report
Strategy
Design
Execute
Evaluate
Who Is Responsible for Validation?
Developer
Business Unit(User)
Quality Assurance
ValidationProcess
Independent Tester
Business Unit Responsibility
Define system requirements
Define data used in the system
Enforce Change Control
Plan & Perform Acceptance Testing
Ensure adequate backup & recovery of data is available
Work with developers to coordinate schedules
Investigate Defects Develop User
Documentation Develop User
Procedures Enforce security Provide audit trail Train personnel
Developer Responsibility
Follow development process
Work with QA to improve the process
Manage changes to system components
Provide adequate backup & recovery
Maintain configuration management
Plan & Perform Unit Testing
Provide Technical Documentation
Maintain communication with QA , Testers & Business
QA Responsibility
Define Development & Validation Process
Verify personnel have adequate training & resources for their job
Review documentation - adherence to standards
Periodically review process activities
Evaluate vendor capabilities & processes
Verify enhancements & maintenance are documented
Conduct Risk Assessments
Verify business unit has established procedures
Ensure system is adequately secure
Independent Tester Responsibility
Follow Validation Process
Working with Users &/or BA’s, perform Requirements Validation
Work with QA to improve the process
Plan & Perform Integration & System Testing
Conduct Risk Assessments
If required, Plan & Perform Stress & Performance Testing
Production
Testing Levels & Validation
Code
Design
Requirements
Business Needs
Unit Testing
Integration Testing
System Testing
Acceptance Testing
Plan for Maintenance
Prepare for Maintenance during Development
Keep Deliverables and Documentation up-to-date—Requirements—Design—Test Cases, Test Data, Test Results
Maintain all related deliverables during error correction—Requirement specifications—Program specifications—Design specifications—Defect reports
Maintenance Deliverables
Risk Assessment Software updates Hardware upgrades System documentation Manuals, procedures Revised/New Test Cases Updated Validation Plan
Validation Issues
Responsibility lies with the user—If outsourced, the responsibility is shared
between the outsourcing organization and the user
Develop deliverables as system is maintained
Group maintenance changes together Schedule maintenance in releases Follow defined methodology
Purchased System Responsibilities
Validation Activity
Feasibility Review
Requirements Review
Design Review
Code Review
Unit Testing
Integration Testing
System Testing
Acceptance Testing
Conversion Testing
Maintenance Testing
Vendor
x
x
x
x
x
x
Customer
x
x
x
x
x
x
System Staff Users
x
x
x
x
x
Cost of Quality The Cost of Correction
Reqmts Design Code Test Prod'n
$
$
AppraisalAppraisalPreventionPrevention
FailureFailure
FailureFailureAppraisalAppraisal
PreventionPrevention
WithoutQuality Process
WithQuality Process
Savings
Cost of Quality Real Benefits
Incremental Benefits
Incremental Costs
Time
Development Maintenance
Net Costs
Net Benefits
Benefits
Costs
Validation Costs and Benefits
Self-Assessment
Are risk assessments being performed on projects?
Is there a methodology in place for system development and validation?
Are verification techniques, such as walk-throughs, being used?
Have all personnel been trained in validation techniques?
Are standards in place for system development and validation?
Are the users active participants in the validation process?
Are the results of the validation used to improve the development and validation processes?
Is the culture ready to accept new processes for validation?
Is there someone responsible for validation of systems?
Y N ?
Validation:Did we build the right system?
Testing:Did we build the system right?
System Validation
Validates BOTH Business & Technical Requirements
Reduces Development Costs by identifying Errors early in Development Cycle
Reduces Risk of Operating Failures after Implementation
Reduces Post Implementation Development Costs by Fixing Defects during Development
Improves Overall Quality of System Greatly improves User Satisfaction
Projects Delivered Without Validation
Projects Delivered With Validation
The Road to Success!!
Summary
Questions / Discussion