Identifying and Using a Project’s Key Subprocess

Metrics

Jeff S. HolmesBTS Fort Worth

Everyone Loves a Hero

Heroes Come Through!

Firemen – Saves Baby in Burning House Policemen – Catches Bad Guy Athlete – Hits Game Winning Homerun

Software Engineering Heroes!

All night coding! Debugging over the weekend!

THIS SHOULD NOT BE THE NORM!

How To Minimize “Fire Drills”?

Preventative Actions Proper wiring can prevent fires. Locking your car can prevent theft. Don’t get behind in the ball game. Understand project status earlier.

Metrics, metrics, everywhere…

But What is Really But What is Really Important?Important? Customer WantsCustomer Wants

– FunctionalityFunctionality– Zero DefectsZero Defects– On TimeOn Time

What software What software metrics map to metrics map to these?these?

How can we optimize How can we optimize these outputs?these outputs?

BTS Fort Worth Approach

Selected DMAIC to Improve Process Identified Project with Two Years of Data Performed Statistical Analysis Conducted Pilot Currently in “Control” Phase

DMAIC : Define

Identify “what is important” BTS FW Monitors

Productivity (KLOC/Hour)*

Quality (Post Release Defects/KLOC)Schedule Adherence

These are BTS FW “Big X’s”

DMAIC : Measure

Software Development Life CycleRequirementsResources

Perfect Software!

The “Simple” View

DMAIC : More DetailsRequirements

RequirementsResources

Perfect Software!

Design

Code

Test

Perfect RequirementsResources

Perfect Design Perfect Models

Perfect DesignPerfect ModelsResources

Perfect Code

Perfect CodeResources

DMAIC : Subprocess Identification

BTS FW Identified Following Subprocesses Planning Phase Requirements Phase Design Phase Code Phase Test Phase Release Phase Code Inspections

DMAIC : Measured Data BTS FW Uses Following Data:

# Requirements # Developers on the project (Resources) % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection

DMAIC : Data Sources BTS FW Data

DOORs for Requirements Project Plans # Developers and % Times ClearCase for Code Size Inspection Database for DDR

DMAIC : BTS Subprocess Metrics Planning

Design

Code

Test

Req CountTime in Phase

Req ChurnTime in Phase

Requirements

ResourcesTime in Phase

Req ChurnKLOC

Time in Phase

Time in Phase

Release Time in Phase

Code Inspections DDR

DMAIC : BTS Subprocess Metrics Planning

Design

Code

Test

Req CountTime in Phase

Req ChurnTime in Phase

Requirements

ResourcesTime in Phase

Req ChurnKLOC

Time in Phase

Time in Phase

Release Time in Phase

Code Inspections DDR

Productivity (KLOC/Hour)

Quality (PR Defects/KLOC)

Schedule Adherence

?

?

?

?

?

DMAIC : BTS FW Analysis Project Data

• 8 Releases since 2002• Similar work• “Stable” team

Used Step-wise Linear Regression to • Identify statistically significant factors• Develop prediction formulas for “Big X’s”

CAUTION !!

The following slides contain statistics that could be hazardous to your health!

Persons who suffer from narcolepsy or “statisticitis” should consider leaving the room.

DMAIC : Stepwise Linear Regression Describes the relationship between one

'predicted' variable and 'predictor' variables

Goal – get the simplest equation with the best predictive power for• Productivity – KLOC/Hour• Quality – Post Release Defect/KLOC

DMAIC : Standard Least Squares

Model accounts for99.82% of variance.

DMAIC : Significant Effects

Most significant effects• % Time in Req• Average DDR• Interaction between % Time in Code and Average DDR

< 0.05 is significant

DMAIC : Standard Least Squares

Model accounts for90.62% of variance.

DMAIC : Significant Effects

Most significant effects• % Time in Requirements• Interaction between % Time in Requirements

and Requirements Churn

< 0.05 is significant

DMAIC : Statistically Significant # Requirements # Developers on the project % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection

DMAIC : Key Subprocess Metrics Planning

Design

Code

Test

Req CountTime in Phase

Req ChurnTime in Phase

Requirements

ResourcesTime in Phase

Req ChurnKLOC

Time in Phase

Time in Phase

Release Time in Phase

Code Inspections DDR

Productivity (KLOC/Hour)

Quality (PR Defects/KLOC)

Schedule Adherence

?

?

?

?

?

DMAIC : Variation Analysis Prediction formulas generated to identify:

• Good and bad variance• Most significant factors

NOTE: Prediction formula uses all effects from the models, not just the significant ones. Formula added in Percent Planning.

DMAIC : Factor WeightingMetric 10% Deviation in Each Subprocess Area% Plan 10% 0 0 0 0

% Req 0 10% 0 0 0

% Code 0 0 10% 0 0

Req Churn 0 0 0 10% 0

Avg DDR 0 0 0 0 10%

Productivity 0 3% 31% 0 9%

Quality 14% 40% 0 6% 0

Cycle Time 0 3% 24% 0 9%

DMAIC : Factors’ Effects

Metric LOC/Hr Effect PR Defect/KLOC Effect% Planning Time N/A More time = Less

defects/KLOC

% Req Time More Time = Less KLOC/Hr

Less time = More defects/KLOC

% Code Time More Time = Less KLOC/Hr

N/A

Code Insp. DDR Higher DDR = Less KLOC/Hr

N/A

2SR Req. Churn N/A Less Churn = Less defects/KLOC

DMAIC : BTS FW Limits

Green limit indicates direction a metric can deviatefrom the average and have desired results.Red indicates direction of undesired results.

Metric Average Std Dev Lower Limit

Upper Limit

% Planning Time 18 % 8 % 10 % 26 %

% Req Time 15 % 8 % 7 % 23 %

% Code Time 21 % 12 % 11 % 33 %

Code Insp. DDR 1.89 .5 1.39 2.39

2SR Req. Churn 1.76 1.89 0 3.65

Subprocess Metrics Notes Initial Data Left Much to be “Desired” Despite Poor Data, the Analysis

Identified:• Which Metrics and Processes Are Significant• Prediction Formulas Based on Project’s Data• Insight into Factors’ Effects• Limits for Monitoring the Factors

DMAIC : Pilot Confirmation Used prediction formulas on other projects Compared project actuals vs. predicted. Used historical data from 5 projects. Unable to compare predicted quality versus

actual. (Predicted LOC/Hr)• These projects have not been in field long enough for

CRUD to stabilize. Interesting results found on predicted LOC/Hr.

Project A Project B Project C Project D Project E% Plan 0% 18% -65% 10% 17%% Req -28% 10% -97% 4% 17%% Code -348% 20% 0% 43% 25%Inspection DDR 1.2 1.2 0 1.51 1.89Requirements Churn-2SR 1 1.81 0 0 0Pred LOC/Hr 92.673688 4.789660 25.540537 2.288774 2.418825Pred PR Def/KLOC 7.7196696 1.33169549 28.1019536 3.39281 0.4010827

Actual LOC/Hr 1.21 1.79 0.829 2.039 1.945LOC/Hr Delta 99% 63% 97% 11% 20%

DMAIC : Predicted vs. Actual LOC/Hr

•Projects A, B, and C projects had huge deviations.• Projects D and E were within 20%.

DMAIC : Improve Performance? So what? How do you use this information? Does Project Management have confidence

in this analysis?

More emphasis on statistically significant activities

Resulting in Increased Productivity

On-Time Delivery Desired Functionality Delivered

Improved Quality

DMAIC : Applying Analysis

DMAIC : Agile Processes BTS FW Adopted Agile Practices

Iterative Development Prioritizes Requirements Negates Requirements Churn

Pair Programming Optimizes Coding and Inspection Time

Minimal Documentation Moves effort from non-statistical activities.

DMAIC : Agile Pilot Results Productivity

0.00291 KLOC/Hr 20% improvement from 0.002399

Inspection Defect Detection Rate 1.18 Defects/Hr Detected 48% improvement from 0.8

Quality 0 Post Release Defects!

DMAIC : Agile Pilot Results Customer Wants

Functionality – All functionality delivered Zero Defects – No customer found defects On Time – Product delivered 6 months early!

DMAIC : Agile Monitoring Monitor Iterations Not Phases Refactoring Subprocess Monitoring Two Agile projects in-work now

DMAIC : Agile Monitoring Monitoring

LOC per week Defects caught per week by inspection Defects caught per week by test Time spent per week Ratio of new work to correction work.

DMAIC : Agile Metrics% Fault Work

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

7/31/2005 8/7/2005 8/14/2005 8/21/2005 8/28/2005

SummarySummary Save your “heroes” for real crises. Save your “heroes” for real crises. Understand subprocesses Understand subprocesses Monitor subprocessesMonitor subprocesses Seek to optimize key subprocessesSeek to optimize key subprocesses

RecommendationsRecommendations Examine current project data, it Examine current project data, it

could prove to be very valuable!could prove to be very valuable! Improve data capture on important Improve data capture on important

data.data. Use the data as a guideline, but Use the data as a guideline, but

experience can never be discounted.experience can never be discounted.

THANK YOU!THANK YOU!Jeff S. HolmesJeff S. Holmes

Principal Staff Software EngineerPrincipal Staff Software EngineerMotorola Six Sigma Black BeltMotorola Six Sigma Black Belt

Fort Worth BTS Development TeamFort Worth BTS Development TeamFort Worth, TXFort Worth, TX817-245-7053817-245-7053

J.Holmes@Motorola.comJ.Holmes@Motorola.com