C S E Quality Management –Lessons of COQUALMO ...sunset.usc.edu/classes/cs510_2011/ECs_2011/EC-11b... · C S E USC Quality Management –Lessons of ... PCAP PCON TOOL PREC PVOL

C S E

USC

Quality Management – Lessons of

COQUALMO

(COnstructive QUALity MOdel)A Software Defect Density Prediction Model

Copyright 1999-2007 USC-CSSE 1

AWBrown and Sunita Chulani, Ph.D.

{AWBrown, sdevnani}@CSSE.usc.edu}

USC-Center for Systems & Software Engineering (USC-CSSE)

C S E

USC

OutlineBehavioral Underpinnings

• Hidden Factory

• Defect Types

COQUALMO Framework

• The Defect Introduction Sub-Model



– Expert-Judgment Model + Some Initial Data Results

• The Defect Removal Sub-Model

– Expert-Judgment Model

(Result of COQUALMO Workshop)

• COQUALMO Integrated with COCOMO II

C S E

USC

USC Modeling Methodology


C S E

USC

Wideband Delphi

• Operationally

• Final values (for each parameter)

Max=(Hmax + 4*AVEmax + Lmax)/6


C S E

USC

Fig 11., Pg 170 SwCEwCII


C S E

USC

Model Framework

• • •

Residual

Software

DefectsCode Defects

Requirements Defects

Design Defects

Defect Introduction pipes


Defect Removal pipes

C S E

USC

Software Devel. Process(es) (cont.): the hidden factory


C S E

USC

Outline

• Model Framework

�The Defect Introduction Sub-Model

– Expert-Judgment Model + Some Initial Data

Results




– Expert-Judgment Model (Result of

COQUALMO Workshop)


• Future Plans

– Call for Data

C S E

USC

The Defect Introduction (DI)

Sub-Model

Software Size estimate

Software product, Defect

Number of non-trivial

requirements, design and

code defects introduced


Software product,

process, computer and

personnel attributes

(subset of COCOMO II

factors)

Defect

Introduction

Sub-Model

code defects introduced

C S E

USCA-Priori Expert-Judgment

Based Code DI Ranges

SITE

DOCU

PEXP

AEXP

ACAP

DATA

TIME

STOR

RUSE

FLEX


1.00 1.50 2.00 2.50 3.00

PMAT

RELY

RESL

CPLX

PCAP

PCON

TOOL

PREC

PVOL

LTEX

SCED

TEAM

SITE

C S E

USC

DI Model Equations

j 1

3

j

B

iji 1

21

A (Size) (DI driver)j

= =

∑ ∏∗ ∗ −

•Estimated Number of Defects Introduced =

� j identifies the 3 artifact types (requirements, design and coding).

� A is the multiplicative calibration constant.

� B is initially set to 1

� (DI-driver) is the Defect Introduction driver for the jth artifact and the i

th factor.


•For each artifact j, Quality Adjustment Factor (QAF)

QAF DIR - driverij

i 1

22

j =

=

∏

� (DI-driver)ij is the Defect Introduction driver for the jth artifact and the i

th factor.

j 1

3

j

B

jA (Size) QAFj

=

∑ ∗ ∗

•Estimated Number of Defects Introduced =

C S E

USC

Initial Data Analysis on the DI Model

Type of

Artifact

1970Baseline

DIRs

Quality

Adjustment

Factor

Predicted

DIR

Actual DIR Calibrated

Constant

(A)

1990Baseline

DIRs

Requirements 5 0.5 2.5 4.5 1.8 9


Requirements 5 0.5 2.5 4.5 1.8 9

Design 25 0.44 11 8.4 0.77 19

Code 15 0.5 7.5 16.6 2.21 33

DIR = Defect Introduction Rate

C S E

USC

Outline

• Model Framework



Results

�The Defect Removal Sub-Model


�The Defect Removal Sub-Model


COQUALMO Workshop)


• Future Plans

– Call for Data

C S E

USC

Defect

Number of non-trivial

requirements, design and

coding defects introduced

Number of residual

The Defect Removal (DR) Sub-Model


Defect Removal

Sub-Model

Defect removal activity

levels

Number of residual

defects/ unit of size

Software Size Estimate

C S E

USC

D e f e c t R e m o v a l P r o f i l e s

• 3 relatively orthogonal profiles

–Automated Analysis

–People Reviews

–Execution Testing and Tools


• Each profile has 6 levels

–Very Low, Low, Nominal,

High, Very High, Extra High

• Very Low--removes the least number of defects

• Extra High--removes the most defects

C S E

USC

Automated AnalysisRating Automated Analysis

Very Low Simple compiler syntax checking.

LowBasic compiler or additional tools capabilities for static module-level code

analysis, and syntax- and type-checking.

Nominal All of the above, plus

Some compiler extensions for static module and inter-module level code

analysis, and syntax- and type-checking.

Basic requirements and design consistency; and traceability checking.

High All of the above, plus

Intermediate-level module and inter-module code syntax and semantic analysis.


Intermediate-level module and inter-module code syntax and semantic analysis.

Simple requirements/design consistency checking across views.

Very High All of the above, plus

More elaborate requirements/design view consistency checking.

Basic distributed-processing and temporal analysis, model checking, symbolic

execution.

Extra High All of the above, plus

Formalized* specification and verification.

Advanced distributed-processing and temporal analysis, model checking,

symbolic execution.

*Consistency-checkable pre-conditions and post-conditions, but not

mathematical theorems.

C S E

USC

Static [Module-Level Code] Analysis

Wikipedia

• Static code analysis is the analysis of computer software that is

performed without actually executing programs built from that

software (analysis performed on executing programs is known as

dynamic analysis). In most cases the analysis is performed on


dynamic analysis). In most cases the analysis is performed on

some version of the source code and in the other cases some form

of the object code.

C S E

USC

Static [Module-Level Code] Analysis• SWEBOK [sans references]

4.2. Quality Analysis and Evaluation Techniques Various tools and

techniques can help ensure a software design’s quality.

• Software design reviews: informal or semiformal, often group-based,

techniques to verify and ensure the quality of design artifacts (for


techniques to verify and ensure the quality of design artifacts (for

example, architecture reviews, design reviews and inspections,

scenario-based techniques, requirements tracing)

• Static analysis: formal or semiformal static (nonexecutable) analysis

that can be used to evaluate a design (for example, fault-tree analysis or

automated cross-checking)

• Simulation and prototyping: dynamic techniques to evaluate a design

(for example, performance simulation or feasibility prototype)

C S E

USC

Peer Reviews

Rating Peer Reviews

Very Low No people reviews.

Low Ad-hoc informal walkthroughs

Minimal preparation, no follow-up.

Nominal Well-defined sequence of preparation, review, minimal follow-up.

Informal review roles and procedures.

High Formal review roles and procedures applied to all products using basic

checklists*, follow up.


checklists*, follow up.

Very

High

Formal review roles and procedures applied to all product artifacts &

changes; formal change control boards.

Basic review checklists, root cause analysis.

Use of historical data on inspection rate, preparation rate, fault density.

Extra

High

Formal review roles and procedures for fixes, change control.

Extensive review checklists, root cause analysis.

Continuous review process improvement.

User/Customer involvement, Statistical Process Control.

* Checklists are lists of things to look for or check against (e.g. exit criteria)

C S E

USC

Syntactic Versus Semantic Checking

Both sentences below are semantically correct,

only one is semantically correct.

• A panda enters the bar, eats shoots and leaves.

• A panda enters the bar, eats, shoots and leaves.


• A panda enters the bar, eats, shoots and leaves.

C S E

USC

Execution Testing and Tools

Rating Execution Testing and Tools

Very Low No testing.

Low Ad-hoc testing and debugging.

Basic text-based debugger

Nominal Basic unit test, integration test, system test process.

Basic test data management, problem tracking support.

Test criteria based on checklists.

High Well-defined test sequence tailored to organization (acceptance, alpha, beta,


High Well-defined test sequence tailored to organization (acceptance, alpha, beta,

flight, etc. test).

Basic test coverage tools, test support system.

Basic test process management.

Very

High

More advanced test tools, test data preparation, basic test oracle support,

distributed monitoring and analysis, active assertion checking.

Metrics-based test process management.

Extra

High

Highly advanced tools for test oracles, distributed monitoring and analysis,

assertion checking.

Integration of automated analysis and test tools.

Model-based test process management.

C S E

USC

Technique Selection Guidance

“Under specified conditions, …”

• Peer reviews are more effective than functional

testing for faults of omission and incorrect

specification (UMD, USC)


specification (UMD, USC)

• Functional testing is more effective than reviews

for faults concerning numerical approximations

and control flow (UMD, USC)

C S E

USC

Residual Defects Equation

• Estimated Number of Residual Defects

DResEst, j = Estimated No. of Residual Defects for the jth artifact

DRes C DI (1 DRF )Est, j j Est, j iji

= × × −∏3

i=1


DResEst, j = Estimated No. of Residual Defects for the jth artifact

Cj = Calibration Constant for the jth artifact

DIEst, j = Estimated No. of Defects Introduced for the jth artifact

(output of DI Sub-Model)

i = Defect Removal profile

DRFij= Defect Removal Fraction

C S E

USCDefect Densities from Expert-

Judgment Calibrated COQUALMOAutomated

Analysis DRF

People Reviews

DRF

Execution Testing

and Tools DRF

Product

(1-DRFij)

DI/kSLOC DRes/kSLOC

R 0.00 0.00 0.00 1.00 10 10

D 0.00 0.00 0.00 1.00 20 20

C 0.00 0.00 0.00 1.00 30 30

Very

Low

Total: 60

R 0.00 0.25 0.23 0.58 10 5.8

D 0.00 0.28 0.23 0.55 20 11

C 0.10 0.30 0.38 0.39 30 11.7

Low

Total: 28.5

R 0.10 0.40 0.40 0.32 10 3.2

D 0.13 0.40 0.43 0.3 20 6Nom-

inal


D 0.13 0.40 0.43 0.3 20 6

C 0.20 0.48 0.58 0.17 30 5.1inal

Total: 14.3

R 0.27 0.50 0.50 0.18 10 1.8

D 0.28 0.54 0.54 0.15 20 3

C 0.30 0.60 0.69 0.09 30 2.7

High

Total: 7.5

R 0.34 0.58 0.57 0.14 10 1.4

D 0.44 0.70 0.65 0.06 20 1.2

C 0.48 0.73 0.78 0.03 30 0.9

Very

High

Total: 3.5

R 0.40 0.70 0.60 0.07 10 0.7

D 0.50 0.78 0.70 0.03 20 0.6

C 0.55 0.83 0.88 0.009 30 0.27

Extra

High

Total: 1.57

C S E

USCValidation of

Defect Densities• Average defect density using Jones’ data weighted by CMM

maturity level distribution of 542 organizations is 13.9

defects/kSLOC

• Average defect density using COQUALMO is

14.3 defects/kSLOC

Residual defect density


542 Organizations

0.669

0.196

0.118

0.013 0.004

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Initial Defined Optimizing

% of Organizations

Leading Average Lagging

1.4 7.5 18.3

Residual defect density

from Jones’ data

(.135*1.4 + .196*7.5 + .669*18.3)

= 13.9 defects/kSLOC

C S E

USC

An Independent

Validation Study

• Aim: To validate expert-determined COQUALMO has correct

trends in defect rates

• Sample Project: Size = 110.5 kSLOC

Type of

Artifact

DI DIR Quality Adjustment

Factor (QAFj)

Baseline

DIR

1970’s

Baseline DIR


Artifact Factor (QAFj) DIR Baseline DIR

Reqts 209 2 0.74 3 5

Design 1043 9.5 0.74 13 25

Code 767 7 0.84 8 15

Type of

Artifact

Automated

Analysis

(VL)

Peer

Reviews

(L - VL)

Execution Testing

and Tools (H - VH)

Product

(1-DRF)

DI/

kSLOC

DRes/

kSLOC

Reqts 0 0.13 0.54 0.4 2 0.8

Design 0 0.14 0.61 0.34 9.5 3.23

Code 0 0.15 0.74 0.22 7 1.54

Total: 5.57

Actual Defect Density = 6 defects/kSLOC

C S E

USC

Outline

• Model Framework



Results





COQUALMO Workshop)

�COQUALMO Integrated with COCOMO II

C S E

USC

Integrated COQUALMO

COCOMO II

COQUALMOSoftware Size estimate

Software platform,

project, product and

personnel attributes

Software development effort, cost

and schedule estimate

Defect Introduction

Model


Defect Removal

Model

Defect removal profile

levels

Number of residual defects

Defect density per unit of size

Documents

C S E Quality Management –Lessons of COQUALMO ...sunset.usc.edu/classes/cs510_2011/ECs_2011/EC-11b... · C S E USC Quality Management –Lessons of ... PCAP PCON TOOL PREC PVOL