© Quantitative Software Management, Inc.

Using Quantitative Software Estimation Tools and Techniques to Support Verification and Validation (V&V)

Joseph MaddenQuantitative Software Management, Inc

2000 Corporate Ridge, Suite 700McLean, VA 22102

(703) 790-0055Email: joseph.madden@qsm.com

Web: www.qsm.com

(#2)

• Introduction to Quantitative Software Estimation• Using Quantitative Software Estimation to

Support V&V Planning & Execution• Example Uses of Quantitative Estimation

Techniques at the FAA• Conclusion

Outline

(#3)

About QSM

• QSM founded in 1978 by Larry Putnam, Sr., one of the original pioneers in quantitative software estimation and author of 4 books, including Measures for Excellence and Five Core Metrics.

• Putnam invented the SLIM quantitative estimation tool and began a benchmark database of historical project data. This database now includes over 10,000 validated projects covering most application domains, industry sectors, and development approaches.

(#4)

Benefits of Quantitative Estimation: A Case Study

Before Using SLIM First Year of Using SLIM

Full Implementation of SLIM

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

$0

$2,000,000

$4,000,000

$6,000,000

$8,000,000

$10,000,000

$12,000,000

$14,000,000

$16,000,000

Large telecommunications supplier with operations in more than 130 countries

% Projects Over Budget and Schedule

Dollars Out of Control

In the year before using SLIM, 10 of 12 projects (83%) exceeded budget and schedule. The cost of this excess was more than $15M. QSM was hired to implement a robust estimation process.

Within the first year of using a SLIM estimation process, the percentage of projects over schedule and budget decreased from 83% to 50%, and excess cost reduced from $15M to $6M. After full implementation of SLIM in the second year, the percentage of projects over schedule and budget dropped to 10% and the cost overruns were less than $500K1.

1The above case study was published in 2003 in the book Five Core Metrics by Lawrence H. Putnam and Ware Myers.

(#5)

The Fundamentals of Quantitative Software Estimation

• Design Complexity• “Software entities are more complex for their size than for perhaps any other human construct.”1

• Organizational Complexity

• Large software projects require a large organization. This organizational complexity “makes overview hard, thus impeding conceptual integrity. It makes it hard to find and control all loose ends. It creates a tremendous learning and understanding burden.”1

• Cost and Schedule Overruns

• The Standish Group reported that 80% of systems are delivered late and over budget and 40% of projects fail or are abandoned.2

• QSM benchmark database shows 70% of projects exceeded budgets and 81% did not meet schedules.3

1 Frederick P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering,” Computer, April 1987.

2 Standish Group. Computer, April 2003. 3 QSM, Inc., Benchmark Database, as of April 2007.

The ProblemSoftware Development is Difficult to Estimate

(#6)

Variable Relationships

• Early software estimation attempts

incorrectly assumed a linear

relationship between variables

• Software size was divided by average productivity rate. Effort was divided by manpower to determine time. Manpower was

increased until time met delivery date.

• Dr. Frederick Brooks demonstrated in The Mythical Man Month that this

assumption is not true. Manpower and time are not interchangeable.1

• Researchers eventually

discovered a nonlinear

relationship between

variables of interest

• Researchers collected and

analyzed a large body of software project data and discovered most relationships are

nonlinear.

• Variables of interest appeared to be a complex power

function of system attributes.2

1 Frederick P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering,” Computer, April 1987.2 Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle

River, NJ, 1992.

Research Revealed a Nonlinear Relationship Between Variables

(#7)

Patterns in Historical Data

Scatter diagrams used to perform trend line analysis, quantifying relationships.

Linkage established between system size, development time, effort, cost, manpower, productivity, and number of defects.1

1 Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle River, NJ, 1992.

This analysis led to derivation of key computational equation1

Product = Productivity Parameter * (Effort/B)(1/3) * Time(4/3)

+1 Std Dev

Avg

-1 Std Dev

Curve Fitting of Historical Data Resulted in the Discovery of Estimation Equations

(#8)

Estimation Equations

Productivity Parameter: Process productivity parameter for software development organization, obtained by calibration from past projects.

Constant “B”: Special skills factor which is function of size. Increases slowly with size as need for integration, testing, quality assurance, documentation, and management skill grows with increased complexity of large projects.

(Effort/B)(1/3) * Time(4/3) = (Size)/(Productivity Parameter)

(Total Effort)/Time3 = Manpower Buildup Parameter

Source: Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle River, NJ, 1992.

Effort: Person-years of work by all job classifications for the software construction or main-build phase: design, coding, inspection, test, documentation, and supervision.

Time: Elapsed calendar schedule in years for software construction phase.

Manpower Buildup Parameter: Computed by calibration from past projects.

“Putnam Equations”

(#9)

Key Time and Effort Rules

• Small changes in duration

result in large changes in

effort

• The relationship

between effort and

time is exponential:

Effort = Constant/Ti

me4

• Extending development time

from 18 to 19 months (a 5.5%

increase) reduces the effort required

by 19.5%.1

• There is a minimum

development time

• “More software projects

have gone awry for lack of

calendar time than

for all other causes

combined.”2

Log Time

Lo

g E

ffo

rt

(MBI*) (Size & PI*)

ImpossibleRegion Minimum

TimePoint

No data to the left of the MBI line has ever been seen.1

*MBI and PI are indexes based on the Manpower Buildup and Productivity parameters.

1 Lawrence H. Putnam. Measures for Excellence: Reliable Software On Time, Within Budget, Prentice Hall PTR, Upper Saddle River, NJ, 1992.2Frederick P. Brooks, Jr., “No Silver Bullet: Essence and Accidents of Software Engineering,” Computer, April 1987.

(#10)

The Shape of Projects

1960s: Dr. Peter Norden found that engineering projects could be depicted by a Rayleigh Curve to predict staffing1

1970s: Larry Putnam Sr. applied the principles to software projects—this offered a mathematical way of predicting broader project behavior2

Traditional project management often tries (often unsuccessfully) to “level-load” projects

Sta

ffing L

eve

l

Time

Sta

ffing L

eve

l

Time

1. Norden, P.V.; “On the Anatomy of Development Projects,'' IRE Transactions on Engineering Management, 1960 Volume 7, Number 1, pp. 34-42.

2. Putnam, Lawrence and Myers, Ware. Measures for Excellence. Yourdon Press 1992. pp.44-45

(#11)

Reliability Model for Predicting Software Quality

Defe

cts

Time

Defects in software also tend to follow a Rayleigh curve.

We can use this to predict quality in a software product.

SLIM-Estimate uses Mean Time To Defect (MTTD): the average time between defects appearing as a measurable indicator of quality.

No work product (yet) to contain defects

Defects rise as work products are created Late in project

primary task is defect removal

(#12)

Reliability Model Variable Relationships

Size: Historical data has shown that as the code size increases, the number of defects increases. The rate of defect increase is close to linear.

Peak Staffing: Historical data has shown that adding people increases the defect creation process at a rapidly accelerating rate. For example:• A project of 350,000 SLOC using 40 people at peak staff would

create approximately 2,125 total defects. • If 60 people were used, 3 months of schedule compression would

be gained but 3,010 defects would be created.

(#13)

Quantitative Estimation

Software Size:• SLOC• Function Points• Objects, Etc.

Uncertainty

Process Productivity:• Methods/Tools• Tech Complexity• Personnel Profile

Management Constraints:• Max People• Max Budget• Max Schedule• Required Reliability

Optimum Estimate (Max Probability of Meeting Constraints)

Evaluate Practical Alternatives

Generate Plans

Inp

uts

Ou

tpu

ts

Staffing

Cost

Probability

Defects

SLIM-Estimate

(#14)

Inp

uts

Quantitative Control

Aggregate Staffing Rate

0.0

5.0

10.0

15.0

S 87654321S 321

People

3 9 15 21 27 33 *

Size

0

20

40

60

S 87654321S 321

ESLOC (thousands)

3 9 15 21 27 33 *

Total Defect Rate

0

40

80

120

S 87654321S 321

Defects

3 9 15 21 27 33 *

Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes R&D, C&T, T&ES = Start, 1 = PDR, 2 = INC_1, 3 = INC_2, 4 = INC_3, 5 = ST, 6 = TRR, 7 = FAT, 8 = 99R

Tracking of Plan Against Approved Baseline

Monthly Metrics Control Charts

Project ManagementMilestones

Software DevelopmentSize Actuals

Cost AccountingEffort/Cost Actuals

Software TestDefect Actuals

Variance from Plan

Forecast to Complete

Evaluation of Alternative Management Strategies

Ou

tpu

ts

SLIM-Control

(#15)

Quantitative Benchmarking

Construction & Test Trends

MB Duration (Months) vs Effective SLOC

1 10 100 1,000

Effective SLOC (thousands)

1

10

100

MB

Duration (M

onths)

MB Effort (MM) vs Effective SLOC

1 10 100 1,000

Effective SLOC (thousands)

1

10

100

1,000

10,000

MB

Effort (M

M)

PI vs Effective SLOC

1 10 100 1,000

Effective SLOC (thousands)

0

5

10

15

20

25

PI

MB Peak Staff (People) vs Effective SLOC

1 10 100 1,000

Effective SLOC (thousands)

0.1

1

10

100

1,000

MB

Peak S

taff (People)

All Systems Avg. Line Style 1 Sigma Line Style

Shows where benchmarked project lies in relation to historical database of

similar projects.

Data is compared for schedule, effort, productivity, and staffing vs. size along

standard deviation lines.

(#16)

Using Quantitative Software Estimation to Support V&V Planning

and Execution

(#17)

+

Making software schedules and associated V&V activities more predictable

Estimating V&V Effort and Schedule

Within the control of the V&V team:• Duration and effort for

complete inspection or testing cycle

Outside the control of the V&V team:• Delivery date for the

software• Volume of defects and

rework• Number of test cycles

required to reach target quality

Staffing & Probability Analysis - Example engineering project

Avg Staff (people)<Normal Schedule>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Jan'10

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'11

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'12

Feb Mar

0

5

10

15

20

Avg Staff (people)

987654321

R&DC&TREL

Milestones 0 - SRR 1 - PDR 2 - CDR 3 - CUT 4 - IT 5 - SIT 6 - ST 7 - FCS 8 - LP 9 - HVP

Milestones 0 - SRR 1 - PDR 2 - CDR 3 - CUT 4 - IT 5 - SIT 6 - ST 7 - FCS 8 - LP 9 - HVP

RISK GAUGE<Normal Schedule>

Life Duration <= 30 Months Life Cost <= 7000 K $C&T Peak Staff <= 25 people C&T MTTD >= 33.6 Hrs

DurationCost

Peak StaffMTTD

% 0 10 20 30 40 50 60 70 80 90 100

SOLUTION PANEL - <Normal Schedule>

DurationEffortCost

Peak StaffMTTD

Start Date

C&T16.133.1

4252.917.1

32.795/14/2010

Life Cycle25.445.1

5800.217.1

201.081/1/2010

MonthsPHR (K)$ (K)peopleHrs

PI=14.4 MBI=2.1 Eff SLOC=100,000

CONTROL PANEL<Normal Schedule>

PI 11.5 17.3

14.4

Peak Staff 13.7 20.5

17.1

Eff SLOC (K) 80 120

100

Defect Rate Total<Compressed Schedule>

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Apr'10

May Jun Jul Aug Sep Oct Nov Dec Jan'11

Feb Mar Apr May Jun Jul Aug Sep

0

50

100

150

200

250987654321

Project: Example engineering proj...

Quantitative estimation techniques can help predict these items.

Takeaway: Quantitative estimation can help make V&V effort and duration more predictable

Predicted volume of defects

Estimated duration

(#18)

Total Defect Estimate - Example engineering project

MTTD Total (Hrs)<Normal Schedule>

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29Apr'10

May Jun Jul Aug Sep Oct Nov Dec Jan'11

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'12

Feb Mar Apr May Jun

0

400

800

1,200987654321

Defects Remaining Total<Normal Schedule>

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29Apr'10

May Jun Jul Aug Sep Oct Nov Dec Jan'11

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'12

Feb Mar Apr May Jun

0

400

800

1,200987654321

Project: Example engineering proj...

Accurately predicting when software is likely to reach target reliability

Takeaway: This technique was used successfully for reliability analysis of the Voice Switching and Control System (VSCS) software at FAA prior to delivery to ensure it was stable enough to deliver and deploy

(#19)

Errors (SysInt-Del) vs Effective SLOC

10 100 1,000

Effective SLOC (thousands)

10

100

1,000

Current Solution Historical Projects QSM ENGINEERING GROUP Avg. Line Sty le 1 Sigma Line Sty leProject: Example engineering proj...

Understanding whether the number of actual defects discovered compares favorably with

other projects

593 defects at +1σ

199 defects average

68 defects at - 1σ

Takeaway: Statistical benchmark data can be used to help answer the question: are there are too many or too few defects?

(#20)

Examples of Quantitative Software Estimation Techniques at the FAA

(#21)

Example Use of Quantitative Software Estimation Techniques at the FAA

• RRI Program. Independent Estimate and Risk Assessment for the FAA for a new generation air-ground router and communication system being built by five separate contractors (three US, one French, one Irish) building different pieces of the system.

• Terrorist Screening Software. Estimates of schedule cost and risk for each of 9 airlines to build terrorist screening software to use in airport screening departure lounges. Recommended to FAA reasonable costs for each system for negotiation between airline and FAA. Validated estimates using SLIM Metrics and QSM database.

• OASIC. Bid Evaluation of competitors to do an upgrade to a flight planning system. Calibrated historic data, did an analysis of each vendor’s bid and provided some alternatives for FAA to consider.

• URET. Independent estimate of an FAA system prior to award. Calibration data and defect analysis of a historic project by the same vendor was done for tuning purposes.

• STARS. Independent estimate of an airfield radar system for the FAA. Calibration and analysis of a vendor bid prior to award. Emphasis on whether they could do the project in the specified time frame.

• ITWS. Competitive Bid Evaluation of 3 vendors competing for a new weather system for the FAA. Calibration of each vendor’s historic data, a baseline government should cost scenario for comparison, and evaluation of each vendor’s bid compared with the baseline, QSM tend lines, and specially tailored data sample. Determine whether bids were consistent with past performance.

• VSCS. Reliability analysis of the software prior to delivery to ensure it was stable enough to deliver and deploy.

• Expert Witness Case. Served as expert witness in a protest of termination of a contract by FAA against the prime contractor in the building of Radio Control Equipment for air traffic control. FAA asserted that Prime Contractor did not perform. QSM examined the plans and data concerning the software development to compared it with industry norms for real time control software to see if the progress, effort, and defect performance was consistent with others building software of a similar nature. Case was settled out of court about two weeks prior to trial.

• Airfield Radar System. Bid evaluation of contractor’s ability to perform work required on airfield based radar system. Once the project was under way, performed monitoring and control and re-planning.

(#22)

Conclusion

• Quantitative estimation supports more objective management decisions.

• Significant insight can be gained from a few core software metrics: size, duration, staffing and defects.

• Improved predictability makes everyone’s job becomes easier, including V&V staff.

(#23)

Contact

www.qsm.com

Quantitative Software Management, Inc. 2000 Corporate Ridge; Suite 700McLean, VA 22102O 703.790.0055Email: joseph.madden@qsm.com

Joseph MaddenVice President