Test Metrics That Matter - Process Management Group · Title: Microsoft PowerPoint - Test Metrics That Matter QAI 2006 Handout.ppt Author: Burton Greenberg Created Date: 5/1/2008

"Test Metrics That Matter"

Copyright © MMVI by Process Management Group, Ltd. All Rights Reserved. 1

Copyright © MMVI by Process Management Group, Ltd. All Rights Reserved.

PROCESS MANAGEMENT GROUP, LTD.

The IT quality and testing expertswww.pmgltd.com



Test Metrics That Matter

• Presented by:• Burton J Greenberg, CSQA, CSTE• President, Process Management Group, Ltd.

(847) 550-0977

[email protected]





FOUNDATION OF MEANINGFUL METRICS

• Definitions• Processes

• Consistency• Discipline• Persistence



TEST METRICS THAT MATTER Presentation Agenda

• Metrics Overview• Process Overview• Testing Overview

• Test Levels• Test Types

• Test Metrics• Planning• Risk Management• Value Added

• Recap





TEST METRICS

Produce information which provides insight on the status of Testing.

By blending information with historical data, management can adjust staffing, scope, assignments, etc. to meet project objectives.



METRICS

• Metrics Are Used To Tell Us …• Where We Are• Where We’ve Been• Where We’re Going





Time To Perform a Task



SOURCE OF METRICS

• By-Product of “Doing”• Recording of “Measurable” Components

• Not an independent “Metrics” Effort







language

lan·guage (1): the words, their pronunciation, and the

methods of combining them used and understood by a community

(2): a systematic means of communicating ideas or feelings by the use of conventionalized signs, sounds, gestures, or marks having understood meanings





definition

def·i·ni·tion (1): a statement expressing the essential

nature of something (2): a statement of the meaning of a word or

word group or a sign or symbol







PROCESS MANAGEMENT

A Process is:An integrated collection of procedures, tools, and training

Sound Processes Facilitate:• Tracking• Management• Enforcement• Meaningful Metrics



PROCESS IMPROVEMENT OBJECTIVES

Sound Processes:• Increase product quality

and/or• Increase Development Productivity

and/or• Reduce Development Time

and/or• Improve Business Value





PROCESSES & METRICS

• Processes Need to Facilitate the Production of Metrics Which Provides Answers

• Meaningful Metrics Provide Answers to Questions Yet to be Asked



TYPICAL TESTING QUESTIONS

• As Testing Begins• ?

• During Testing• ?

• After Testing• ?





TEST LEVELS

TEST TYPES

TEST PROCESSES



TEST LEVELS

• Test Activities based upon “WHO” does the testing and “WHAT” baseline criteria used • Unit Testing

– Programmer– Program Specs & Detailed Design Document

• Independent/Integration Testing– Independent Tester– Functional Design/Requirements

• User Acceptance Testing (UAT)– User (or User Liaison)– Requirements & Business Processes





TYPES OF TESTING

• Test Activities that Differ Based upon scope and objectives• Functional Testing• Structural Testing• Performance • Load Testing• Stress Testing• Regression Testing• Parallel Testing• Pilot Testing



TEST DEFINITION

TEST EXECUTION

DEFECT RESOLUTION





TEST DEFINITIONDELIVERABLES

• Test Plan• Test Case• Updated Baseline Documents• Status Reports• While

• Facilitating Estimates of Time & Resources• Establishing a Foundation for Test

Management



TEST DEFINITIONPROCESS

• Incremental Development• Define Major Business Functions / System

Components for the Application• Define Major Business Conditions• Document Permutations to Identify “What”

needs to be Tested• Explicit “Actions” and “Expected Results”

using Baseline Requirements





TEST DEFINITIONTEST MANAGEMENT

• Track Progress and Report Status• Test Cases Titled• Test Cases Under Development• Test Cases Completed• Test Cases Approved



ANSWER TO FIRSTKEY QUESTION

• Beginning• How Much Will Testing Cost ?

• Using Estimated Number of Titled Test Cases• Apply Historic Rate of Hours to Develop• Apply Historic Rate of Hours to Execute• Can Translate to Cost





TEST EXECUTIONDELIVERABLES

• Updated Test Plan• Updated Test Case• Defect/Problem Reports• Status Reports



TEST EXECUTION PROCESS

• Test Set up• Execution of setup tests to establish foundation• Test Files restored to proper configuration• Ensure tables / parameters coded for the test

(clearly document content of these entries)

• Test Execution• Initiate test per steps documented in Test Case• Carefully analyze test results vs. pre-defined

expected results





Test ExecutionAssessment of Defects

• Severity Codes• Determination of a defects impact to the application

system– Ranges from ‘cosmetic’ to ‘halts performance’

• Objectively Assigned by the Test Analysts• Priority Codes

• Determination of urgency of resolution of a defect– Ranges from ‘immediate resolution’ to ‘defer for future

consideration’

• Provided by Business Owners



Sample Severity Codes

1. System Inoperable (cannot proceed with any processing/testing)

2. Business Function Disabled (can process/test other functions)

3. Function Not Performing As Defined (can process/test function by work-around or with limitations)

4. Cosmetic Only





Sample Priority Codes

1. Immediate resolution required, emergency correction required

2. Next scheduled test release, must include with pending release by developers

3. Include in next release targeted for production, deemed required for production use

4. Defect to first ‘x’ months in production, can implement without correction, to be included in early maintenance fix

5. Although part of base agreement, defer for future consideration, schedule for future release



TEST EXECUTION TEST MANAGEMENT

• Forward to Development for Resolution• Reviews and Assigns

• Produce regular Test Execution Status• Requirements Coverage• Problem Report Metrics

– Defects by Severity– Defects by Component– Defects by Assignment– Defect Aging– Project Specific & ad hoc





ANSWER TO SECONDKEY QUESTION

• During• Is The Application Ready To Install ?

• Using “Open” Defect Reports• Assess & Assign High Priority Defects• Development Can Estimate Time (cost) to

Completion• Development Can Focus on “Right” Fixes• Facilitates Risk Management



DEFECT MANAGEMENTDELIVERABLES

• Updated Software• Release Notes• Defect/Problem Reports• Updated Test Plan• Updated Test Case• Status Reports





DEFECT RESOLUTIONPROCESS

• Project Leader• Reviews/Updates Priority• Assigns for resolution

• Developer• Address Problem• Unit & String Test• Incorporate with Release Notes

• Test Analyst• Test Specific Updates per Release Notes• Regression Testing• Records Status



ANSWER TO THIRDKEY QUESTION

• After • Was Testing Worth It ?

• Using “Closed” Defect Reports• Apply Industry Costs of Defects• Contrast to Cost of Testing (i.e.

Preventative Costs)





NATIONAL INSTITUTE OF STANDARDS & TECHNOLOGY

(NIST) May, 2002 Report• Buggy software costs users, vendors nearly $60B/Year • Better testing infrastructure could reduce the cost by $22.5

billion, though it wouldn't eliminate all software errorsRelative Costs of Defect Removal

Baziuk Study (1995)

Requirements System Testing

Installation Testing

Acceptance Testing

Operations &

Maintenance Costs 1x 90x 90x~440x 440x 470x~880x

Published in NIST“The Economic Impact of Inadequate Infrastructure for Testing” 5/2002







• Language & Definitions• Metrics• Testing Processes

• Test Definition• Test Execution• Defect Resolution

SUMMARYTest Metrics That Matter



TYPICAL TESTING QUESTIONS

• As Testing Begins• How Much Will Testing Cost ?

• During Testing• Is The Application Ready To Install ?

• After Testing• Was Testing Worth It ?





Burton J [email protected]

(847) 550-0977The IT quality and testing experts

www.pmgltd.com