View
232
Download
0
Category
Preview:
Citation preview
8/14/2019 Statistically controlling the software process
1/25
1
Carnegie Mellon University
Software Engineering Institute
Statistically Controlling the
Software Process
Anita D. Carleton and William A. FloracThe 99 SEI Software Engineering Symposium
September 1, 1999
Software Engineering InstituteCarnegie Mellon University
Pittsburgh, PA 15213-3890
Sponsored by the U.S. Department of Defense
1999 by Carnegie Mellon University
8/14/2019 Statistically controlling the software process
2/25
2
Carnegie Mellon University
Software Engineering Institute
Topics
Introduction
Statistical Thinking and Process Thinking
Understanding Variation
Control Charts
Software Example
10 Steps to Get Started and FAQs
8/14/2019 Statistically controlling the software process
3/25
Carnegie Mellon University
Software Engineering Institute
Process Thinking
Focus on the processes to improve quality and
productivity
Managers must focus on fixing processes, not
blaming people
Management action uses data from the process
to guide decisions
Recognize that variation is present in all
processes and that it is an opportunity for
improvement
8/14/2019 Statistically controlling the software process
4/25
Carnegie Mellon University
Software Engineering Institute
Statistical Thinking
Fundamental axioms
all work is a series of interconnected
processes
all processes are variable
understanding variation is the basis for
management by fact and systematicimprovement
Source: G. Britz, How to Teach Others to Apply Statistical Thinking, Quality Progress, June 1997.
8/14/2019 Statistically controlling the software process
5/25
Carnegie Mellon University
Software Engineering Institute
Understanding Variation
While every process displays variation, some processes
display controlled variation, while others displayuncontrolled variation.
- Walter Shewhart
Common cause or controlled variation - due to normal orinherent activities among the process components
Assignable cause or uncontrolled variation - due to
anomalies in the process
total variation = common cause variation + assignable
cause variation
8/14/2019 Statistically controlling the software process
6/25
6
Carnegie Mellon University
Software Engineering Institute
Process BehaviorVariation andStability
Variation = process noise + process anomalies
Stable process =
Stable process = Controlled process
process anomalies
8/14/2019 Statistically controlling the software process
7/25
7
Carnegie Mellon University
Software Engineering Institute
Statistical Process Control
A phenomenon will be said to be
control led when, through the use ofpast experience, we can predict, atleast within l imits, how thephenomenon may be expected to
vary in the future.
Walter A. Shewhart, 1931
8/14/2019 Statistically controlling the software process
8/25
Carnegie Mellon University
Software Engineering Institute
Why SPC for Software Development?
To understand the reliability of human processes
To establish bounds on management expectations
To understand patterns and causes of variations
To validate metric analysis used to forecast and plan
Source: T. Keller, Applying SPC Techniques to Software Development--A Management Approach, 1999 SEPG Conference.
8/14/2019 Statistically controlling the software process
9/25
Carnegie Mellon University
Software Engineering Institute
Control Charts - 1
Lower Control Limit
(LCL)
Upper Control Limit
(UCL)
Specification
Limits
Limits
Control Limits Determined by Process Performance Measurements(Voice of the process)
Specification Limits Set by customer, engineer, etc.(Voice of the customer)
8/14/2019 Statistically controlling the software process
10/25
Carnegie Mellon University
Software Engineering Institute
Control Charts - 2
UCL
LCL
DefectsPer
KLOC
Voice
of the
Process
Time
Inspection Process Performance
8/14/2019 Statistically controlling the software process
11/25
11
Carnegie Mellon University
Software Engineering Institute
Control chartslet you know whatyour processes cando, so that you canset achievable goals.
They represent the voice of the process.
Control charts provide the evidence of stability that
justifies predicting process performance.
Control charts separate signal from noise, sothat you can recognize a process change when
it occurs.
Why Control Charts?
0 5 10 15 20 25 30
10
14
18
22
26
30
LCL=7.89
CL=20.4
UCL=32.9
Backlog of Unresolved Problem Reports
IndividualValues
8/14/2019 Statistically controlling the software process
12/25
12
Carnegie Mellon University
Software Engineering Institute
Detecting Instabilities and Out-of-Control Situations - 1
To test for instabilities in processes, we examinecontrol charts for instances and patterns that
signal nonrandom behavior.
Values falling outside the control limits andunusual patterns within the running record
suggest that assignable causes exist.
8/14/2019 Statistically controlling the software process
13/25
13
Carnegie Mellon University
Software Engineering Institute
Detecting Instabilities and Out-of-Control Situations - 2
Test 1: A single point falls outside the 3-sigma control limits.
Test 2: At least two of three successive values fall on
the same side of, and more than two sigma
units away from, the center line.
Test 3: At least four out of five successive values fall
on the same side of, and more than one sigma
unit away from, the center line.
Test 4: At least eight successive values fall on the
same side of the center line.
Source: Western Electric Handbook
8/14/2019 Statistically controlling the software process
14/25
Carnegie Mellon University
Software Engineering Institute
0 5 10 15 20 25
-20
-10
0
10
20
30
40
50
60
.
LCL
UCL
Individuals
Subgroup No.
Detecting Instabilities and Out-of-Control Situations - 3
Test 3: 4 out of 5
signals in zone B
Test 1:Single point outside of zone CTest 2:
2 out of three beyond zone B
Test 4: 8 successive pointson same side of
centerline
Zone A
Zone A
Zone B
Zone C
Zone B
Zone C
CL
8/14/2019 Statistically controlling the software process
15/25
Carnegie Mellon University
Software Engineering Institute
Component 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Totals
Defects 12 16 18 32 22 16 23 35 15 27 16 25 20 26 20 23 23 36 22 27 17 471
Defect Type Number of Defects per Type per Component
Function 3 5 4 4 4 3 3 20 4 11 2 3 3 5 3 7 4 5 5 15 2 115
Interface 2 2 4 4 3 4 2 3 3 4 2 3 5 3 3 3 2 16 6 2 4 80
Timing 1 1 0 1 1 0 2 1 0 0 2 0 1 1 1 1 1 0 1 0 0 15
Algorithm 0 0 1 14 2 0 0 0 0 0 0 1 5 2 7 6 5 1 2 0 1 47
Checking 1 1 5 1 7 1 1 2 0 1 6 3 1 12 1 0 2 4 3 5 2 59
Assignment 0 2 0 4 1 2 1 3 2 3 2 8 1 0 2 1 2 1 0 1 1 37
Build/Pkg. 3 1 1 2 1 0 0 4 3 6 1 0 2 1 1 1 3 2 2 2 1 37
Document 2 4 3 2 3 6 14 2 3 2 1 7 2 2 2 4 4 7 3 2 6 81
Example: Summary of Defect TypesFound During Component Inspections
Source: W. Florac and A. Carleton,Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley,
June 1999.
8/14/2019 Statistically controlling the software process
16/25
Carnegie Mellon University
Software Engineering Institute
0 5 10 15 20
25
05101520
303540
45
LCL = 0
CL = 22.4
UCL = 44.9
MovingRange
TotalDefects
Component Number
0 5 10 15 20
0
5
10
15
20
25
30
CL = 8.45
UCL = 27.6
XmR Charts for the Total Numberof Defects Found in Component Inspections
Source: W. Florac and A. Carleton,Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley,
June 1999.
8/14/2019 Statistically controlling the software process
17/25
Carnegie Mellon University
Software Engineering Institute
0
1
2
3
4
5
6
ControlCharts
forIndividualDefect
Types
0 5 10 15 20
0
2
4
6
8
10
12Defect Type = Checking
0 5 10 15 200
5
10
15
20
Defect Type = Function
0
2
4
6
8
10
12
14
1618
0 5 10 15 20
Defect Type = Interface
Defect Type = Algorithm
0 5 10 15 20
0
2
4
6
8
10
12
14
16
0
2
4
6
8
0 5 10 15 20
Defect Type = Assignment
0 5 10 15 20
Defect Type = Build/Pkg.
0
2
4
6
8
10
12
14
16
0 5 10 15 20
Defect Type = Documentation
Defect Type = Timing
0.0
0.5
1.0
1.5
2.0
2.5
0 5 10 15 20
Source: W. Florac and A. Carleton,Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley, June 1999.
8/14/2019 Statistically controlling the software process
18/25
Carnegie Mellon University
Software Engineering Institute
0 5 10 15 200
2
4
6
8
10
12
0
2
4
6
8
10
12
14
16
Defect Type = Timing
0.0
0.5
1.0
1.5
2.0
2.5
0 5 10 15 20
0 5 10 15 200
5
10
15
20Defect Type = Function
X
X
X
0
2
4
6
8
10
12
14
16
18
0 5 10 15 20
Defect Type = Interface X
0 5 10 15 200
2
4
6
8
10
12
14
16
X
Defect Type = Algorithm
X XX
X
0
1
2
3
4
5
6
0 5 10 15 20
Defect Type = Build/Pkg. X
Defect Type = Documentation
0 5 10 15 20
X
0 5 10 15 20
0
2
4
6
8Defect Type = Assignment
X
Defect Type = CheckingX
RevisedControl
Chartsfor Eachof the
DefectTypes
8/14/2019 Statistically controlling the software process
19/25
Carnegie Mellon University
Software Engineering Institute
810121416182022
24
0
2
4
6
810
0 5 10 15 20LCL=6.634
CL=15.81
UCL=24.99
MovingRang
e
TotalDefec
ts
Component Inspection Sequence0 5 10 15 20
CL=3.45
UCL=11.27
Improved Process-Reduction in Defect Insertion
Source: W. Florac and A. Carleton,Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley,June 1999.
8/14/2019 Statistically controlling the software process
20/25
Carnegie Mellon University
Software Engineering Institute
EvaluatingProcessPerformance
8/14/2019 Statistically controlling the software process
21/25
21
Carnegie Mellon University
Software Engineering Institute
10 Steps to Get Started - 11. Get familiar with SPC techniques; refer to the
SEI measurement guidebook and related
books and papers on SPC.
2. Obtain a SPC tool.
3. Identify process issues.
4. Identi fy process performance attributes.
5. Select and define measures.
6. Collect data.
7. Organize the data and ensure principles underlying SPC
hold (e.g., homogeniety)
8/14/2019 Statistically controlling the software process
22/25
22
Carnegie Mellon University
Software Engineering Institute
10 Steps to Get Started - 28. Plot/graph data.
9. Examine each plot/graph for process stability,
process shifts, and assignable causes. (a) If process NOT STABLE, THEN:
- cannot determine the capabil ity of the process
- no basis to predict outcomes
- action: understand why the process is not
stable and determine what steps can be taken
to achieve stability
(b) If process is STABLE, THEN:
- capability of the process can be determined
and used to compare future process performance
- action: predict future performance and/or
examine other process constituents
10. Run additional analysis as situation requires.
8/14/2019 Statistically controlling the software process
23/25
23
Carnegie Mellon University
Software Engineering Institute
FAQs - 1
1. There are many types of control charts--which ones are
relevant to software?
- XmR charts
- Average and Range Charts
- c and u charts
2. Can I construct control charts with limited software data? How
much data is enough?
- Set trial limits as soon as possible (be cautious in
interpreting initial results)
- Can get early indications of presence of
assignable causes
- Concluding a process is stable with < 20 or 30
subgroups is risky
- Update control limits as more data becomes available
8/14/2019 Statistically controlling the software process
24/25
24
Carnegie Mellon University
Software Engineering Institute
FAQs - 2
3. Do I need to be a high maturity organization to apply SPC
techniques?
- it is advisable to have basic measurement practices
defined ini tially and then use SPC to manage and
improve your process performance
4. Where in the software process can I apply SPC?
- Useful for defining process stability for design, code, and
inspection processes
- Useful for determining i f processes are being conducted
as intended
8/14/2019 Statistically controlling the software process
25/25
25
Carnegie Mellon University
Software Engineering Institute
Recommended