A Practical Approach To Defining Useful Measures

Facilitating your process journey …

A Practical Approach to Defining Useful Measures

Houston Texas Chapter of the IIBA

Houston, TX

August 7, 2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC


28/5/2008

Measurement and Analysis Process Overview

Plan Measurement

Perform Measurement

Evaluate Measurement

Agenda



38/5/2008

Core Measurement Activities

Measurement Process Model

Plan Measurement

Perform Measurement


Objectives and Issues

User Feedback

Analysis Results

Measurement Plan

New Issues

Improvement Actions

Analysis Reports and Performance Measures

Technical and Management

Processes



48/5/2008

Aids early problem identification

Supports making informed decisions and justifying the

decisions

Supports tracking of progress to

achieve specific goals and objectives

Supports evaluating process

and project performance

Enables effective communication of

process and project performance

results

Role of Measurement

The purpose of measurement is to provide the proper objective information needed to make informed decisions that affect the business and/or project outcome

Most effective when implemented in support of an organization’s business and technical objectives and when integrated with existing technical and management activities that define a project



58/5/2008

3. Processes define the measures that support the measurement objectives

2. Information needs define the measurement objectives which in turn define the measures that support the information needs

1. Measurement and Analysis is a process

Fundamental Measurement Principles - 1



68/5/2008

6. Measures are used to aid the decision making process

5. Measures are analyzed to ensure that only necessary data are collected to address the measurement objectives

4. Processes are appropriately “instrumented” to allow data collection




78/5/2008

9. Measurement and Analysis techniques are applied to the Measurement and Analysis process

8. Measurements provide objective information to facilitate effective communication

7. Analysis results are interpreted within the context of other information and data (integrated analysis)




88/5/2008

Core Measurement Activities

Measurement and Analysis Process

Plan Measurement

Perform Measurement


Objectives and Issues

User Feedback

Analysis Results

Measurement Plan

New Issues

Improvement Actions

Analysis Reports and Performance Measures

Technical and Management

Processes

Principle #1Measurement and Analysis is a process



98/5/2008

Information Needs

Objectives, Issues

Measures

Issue-Driven Measurement

Principle #2Information needs define the measurement objectives which in turn define the measures that support the information needs



108/5/2008

Measurement Framework

A B C

Measurement Repository•Estimates•Quality data•Peer Review data•Customer Satisfaction data•Etc.

Project 1 Project 2 Project 3 Project 4

Goals and Objectives

Performance Measures

Process Measures

Process Data

Historical Measures

Project /Product Data

Project /Product Focus

Process Focus

Corporate Focus

CustomersSenior Management



118/5/2008

Process characteristics will drive: what can be measured

how measures are generated and captured

Measurement should be a by-product of process execution

Measurement and the Process

Coding Activity

1. lines of code2. complexity

Example BaseMeasures

1. # of defects found2. defect type

Testing Activity

Principle #3Processes define the measures that support the measurement objectives



128/5/2008


Plan Measurement

Perform Measurement


Agenda



138/5/2008

Plan Measurement

Identify and prioritize

objectives/issues

Describe information needs

and associated measurable

concepts

Design the measures to address each

information need

Develop the measurement procedures to

collect and organize the data

Build the measurement plan,

integrating measurement into project processes



148/5/2008

Each project and organization has unique information needs

When identifying issues, consider sources such as constraints and assumptions

project or organization characteristics• domain, technology

• people and skills

• product features and acceptance criteria

• cost, schedule, and milestones

risks

past experience of the team

Identify and Prioritize Objectives/Issues



158/5/2008

For each top rated objective or issue, describe the information need who needs to know what information in order to make what decision

Results from measurement must be useable by the person or group (who) answer the need (know what information) enable the action (make what decision)

Describe a measurable concept for each need what data might be used gathered and analyzed at what times reported in what ways

Describe the Information Need



168/5/2008

Issue: Project schedule appears to be very aggressiveInformation Needs:

The Project Manager and team members need to regularly determine which tasks are late, to be able to make corrections in the plan.

The Project Manager needs to know the cumulative effects of task slippage during design, code, and test, to know when a significant problem occurs.

Example of a Common Information Need

Collect and analyze data at a level of detail sufficient to identify and isolate problems.



178/5/2008

Objective Requirements need to be managed better since volatile requirements are causing projects to fail.

Information Need Measurable Concept

Example: The Project Manager needs to know the number of requirements changes that the project can handle without additional schedule or budget increases.

Example: At the end of the week, the Requirements Analyst reports to the Project Manager the current requirements stability, total and by type of change. The Project Manager investigates the requirements growth for the week and the month and modifies the plan as appropriate.

AS TEAMS, DEVELOP AN EXAMPLE INFORMATION NEED AND CONCEPT USING

YOUR HANDOUT

Objectives and Information Needs



188/5/2008

Measurement construct captures a chosen approach identifies levels of measures that address the information need

Levels of a Measurement Construct

one or more collections of Measures are used to build an Information Product

Attribute

Base Measure

Derived Measure

Indicator

Information Product



198/5/2008

History, Skills

Leverage Existing Systems

Measurement Repository

Estimation ToolProject

ManagementSystem

RequirementsManagement

System

TimeReporting

System

DefectReporting

System

ConfigurationManagement

System

FinancialSystem

ProjectABC

Effort, Schedule, Plans, & Actuals

Effort/ Schedule Estimates

Effort Actuals

Milestones, Tasks, Assignments

StatusSizing Estimates, Actuals, Characteristics

Counts, Effort

Defects, Rework

Module and Change Size

Budget, Expenditures

Status

Time Spent

Labor Costs

Cost, Budget, & Actuals



208/5/2008

Tools Available?

Information Needs It Now Serves

Additional Needs it Might Serve

Estimation Models/Methods

Requirements Management

Time Reporting

Project Management

Cost Accounting

Defect Tracking

Code Counting and/or Code Analysis

Configuration Management

Test Automation

Others?

What Can You Leverage?



218/5/2008

Build into process

documenta-tion

Build necessary collection

and analysis tools

Identify process

owner for collection

and analysis

Provide training

Use already established

meetings and briefings

to present and discuss

results

Use Quality Assurance function to

verify process is followed

Institutional-ize

collection, analysis, and

use

Integrate Measurement with the Process

Principle #4Processes are appropriately “instrumented” to allow data collection



228/5/2008


Plan Measurement

Perform Measurement


Agenda



238/5/2008

Perform Measurement

These steps occur continuously throughout a project.

Collect and process base and derived

measures

Generate indicators

Analyze Data

Make decisions and take action



248/5/2008

Data need to be captured as close to their creation point as possible

Raw data need to be verified before analysis

Indicators are the building blocks of analysis

Analysis is an investigative activity - indicator generation should be flexible enough to satisfy known information needs and to dynamically uncover

root causes of problems

All analysis involves comparing plan/baseline/expected with actual results

Key Concepts



258/5/2008

At the reporting periods planned, generate and report the indicators

produce standard indicators on regular basis, as planned; special indicators as needed

analyze indicators to localize problems

identify new indicators that respond to new questions

collect sets of related indicators

Indicators

0

20

40

60

80

100

120

140

# o

f U

nit

s C

om

ple

tin

g U

nit

Tes

t

Project Status Date

Unit Test Status

Plan

Actual

Project: OLDMS

Date Prepared: 1 Oct 2009



268/5/2008

Analyze the indicators systematically, to assess the status of known issues and to identify new issues

Have the analysis done by someone independent of those who produce and use the data

While doing the analysis

identify problems

assess problem impact

forecast outcome

evaluate alternatives

Analyze Data

Principle #5Measures are analyzed to ensure that only necessary data are collected to address the measurement objectives



278/5/2008

Integrated Analysis Example for the Information Need:Are we ready to deliver?

Principle #6Measures are used to aid the decision making process

0

200

400

600

800

Nu

mb

er o

f R

eq

uir

em

en

ts

Project Status Date

Requirements Stability

Plan

Actual

Project: OLDMS


0100200300400500600700

Nu

mb

er o

f P

rob

lem

Re

po

rts

Project Status Date

Problem Report Status

Discovered

Closed

Project: OLDMS


020406080

100120140

# o

f U

nit

s C

om

ple

ted

Project Status Date

Design Progress

Plan

Actual

Project: OLDMS Date Prepared: 1 Oct 2009



288/5/2008

Problem

• Test planning not started

• Development is slightly behind schedule with Component C

• Staffing is below planned levels

Impact

• Test Plan - Medium

• Development - Low

• Staffing - High

Outcome

• Testing will be late or quality will suffer if planning doesn’t start in the next 2 weeks

• If staffing lag continues or worsens, project will be late

Alternatives

• Shift Test Planning to Analysts

• Test Component C last

• Extend schedule

• Limit staff’s non-project-related commitments

Use a systematic analysis process as the basis for reporting results: identify problems assess problem impact forecast outcome evaluate alternatives

Example:

Reporting Results

Principle #7Analysis results are interpreted within the context of other information and data (integrated analysis)



298/5/2008

The measurement results must be clearlyunderstood by the decision makers

Make results available openly, to all those affected

Action must be taken to realize any benefit from measurement

Desired actions may not be possible - may have to optimize within existing constraints

Make Decisions and Take ActionPrinciple #8Measurements provide objective information to facilitate effective communication

Principle #9Measurement and Analysis techniques are applied to the Measurement and Analysis process



308/5/2008


Plan Measurement

Perform Measurement


Agenda



318/5/2008

Evaluating Measurement

Like the products and processes it serves, the measurement process periodically needs review and analysis, too

Evaluate the measuresEvaluate the

measurement process

Update the experience base

Identify and implement improvements



328/5/2008

• identify key issues and information needs

• specify measures that assist decision-making

Planning Measurement

• collect measures and verify the data

• analyze the data and generate indicators

• use the measurement results for decision-making

Performing Measurement

• review the products and process of measurement, to identify improvements

Evaluating Measurement

Summary



338/5/2008

Questions



348/5/2008

Consulting

• CMMI

• Software Engineering

• Systems Engineering

• Process Improvement

Appraising

• SCAMPI A, B, C

• Gap Analysis

Training

• CMMI/Process Improvement

• Action Planning Workshops

• Measurement and Analysis

• Process Area Specific Training

PPQC Services



358/5/2008

Address: Henry Schneider

President/Senior Principal Consultant

Process and Product Quality Consulting, LLC

2111 Heather Green

Houston, TX USA 77062

Phone: 281-218-6682

E-mail: [email protected]

Web Site: www.ppqc.net

Contact Information

mailto:[email protected]

http://www.ppqc.net/

Business

A Practical Approach To Defining Useful Measures