Embed Size (px)
Citation preview
Software Defect PreventionUsing Orthogonal Defect Classification
Twin-SPIN
January 6, 2005
Presented by:
Megan Graham, ASQ CSQE
2
Overview
• Defect prevention is a process used to improve software quality
• Orthogonal Defect Classification is a tool that characterizes defect data used in defect analysis
3
Agenda
• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary
4
Agenda
• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary
5
Defining Defect Prevention
• What does it mean to you?• Defect prevention is a process whose
purpose is to:– identify the common causes of defects, and– change the relevant process(es) to prevent that
type of defect from recurring. (SEI)
• Take what we already know and apply it to what we think we know to produce quality software.
6
A Bug’s LifeRequirementsRequirements
DesignDesign
CodeCode
TestTest
7
A Bird’s Eye View of DP
DecreaseDecreaseDefectsDefectsInjectedInjected
IncreasIncreasee
DefectsDefectsDetecteDetecte
dd
DefectDefectPreventionPrevention
HistoricalHistoricalProjectProjectDefectsDefects
CurrentCurrentProjectProjectDefectsDefects
8
Applying Defect Prevention
• A defect in the software is also a defect in the process (injection and/or detection)
• For DP to work, we need to turn software defects into actionable process defects
Softwaredefects
Processdefects
DP
9
And then there was ODC…
• Orthogonal Defect Classification
• Developed at IBM in the 1990s by Ram Chillarege
• Methodology to characterize software defects and translate into process defects
10
Agenda
• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary
11
Simple ODC Classification Scheme
DefectDefect
TypeTypeTypeType
TriggerTriggerTriggerTrigger
12
Defect Types• FUNCTION: Affects significant capability, end-user interfaces, product
interfaces, interface with hardware architecture or global data structure(s).
• LOGIC: Affects the functionality of a code module and can be fixed by re-implementing an algorithm or local data structure without a need for requesting a high level design change.
• INTERFACE: Affects the interaction of components via macros, call statements and/or parameters.
• CHECKING: Affects program logic that would properly validate data and values before they are stored or used in computation.
• ASSIGNMENT: Requires change in a few lines of code, such as initialization of control blocks or data structures.
• TIMING/SERIALIZATION: Affects the proper management of shared and real-time resources.
• BUILD/PACKAGE/MERGE: Affects product version or configuration; requires a formal changes to reconfigure or rebuild the product.
13
Defect Triggers
Fault Failure
Trigger 1
Trigger 2
Trigger n
14
Review/Inspection Triggers• DESIGN CONFORMANCE: Comparing the implemented design against a
reference –design document, pattern, or guideline.• LOGIC/FLOW: Checking for correctness or flaws using knowledge of the
practice.• BACKWARD COMPATIBILITY: Examining compatibility with prior version of
the product.• LATERAL COMPATIBILITY: Examining for compatibility with other products
and platforms that need to work with this release.• CONCURRENCY: Serialization, shared resources, multi-threaded tasks, timing,
etc.• INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the same
work product.• LANGUAGE DEPENDENCY: Programming standards, specific implementation
considerations, environment restrictions, execution modes, etc.• SIDE EFFECTS: Usage behavior beyond design, but relevant in context. Do A; B
happens.• RARE SITUATION: Unusual issues related to idiosyncrasy of environment,
hardware, or software.
15
Function Test Triggers• SIMPLE PATH: White box – Straightforward usage of code path and
branches.
• COMPLEX PATH: White box – Exercising conditionals, and circuitous coverage.
• COVERAGE: Black box – Straightforward usage of function or single parameterized execution.
• VARIATION: Black box – Straightforward like coverage, but with a variety of parameters.
• SEQUENCING: Black box – Multiple functions, with a specific sequence (order is important).
• INTERACTION: Black box – When two or more bodies of code are involved (order is not important).
16
System Test Triggers• WORKLOAD STRESS: Pushing the limits of performance, resources, users,
queues, traffic, etc.
• RECOVERY: Invoke exception handling, recovery, termination, error percolation, etc.
• STARTUP/RESTART: Major events of turning on, off, or changing the degree of service availability.
• HARDWARE CONFIGURATION: Issues surfaced as a consequence of changes in hardware setup.
• SOFTWARE CONFIGURATION: Issues surfaced as a consequence of changes in software setup.
• BLOCKED TEST/NORMAL MODE: Test could not run during System Test, or customer found nonspecific trigger. Look for additional trigger.
17
Opener
Closer
ODC v5.11 Classification Scheme
Source: ODC v5.11, IBM Center for Software Engineering, www.research.ibm.com/softeng
DefectDefect
TypeType SourceSource
ImpactImpactTriggerTrigger
ActivityActivity
TargetTarget
QualifierQualifier
AgeAge
18
Agenda
• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary
19
When to Apply DP
• DP can be applied to one or more phases of the software lifecycle
• Depends on maturity, goals, etc.
RQTS DSGN CODE TEST
Defect Prevention
Field
Ana
lysi
s Action
Ana
lysi
s Action
Ana
lysi
s Action
Ana
lysi
s Action
Ana
lysi
s Action
20
Defect Analysis
Orthogonal Defect Orthogonal Defect ClassificationClassification
What are the attributes of the defects?
Defect CausalDefect CausalAnalysisAnalysis
When are defectsbeing injected?
Defect TriggerDefect TriggerAnalysisAnalysis
How are defectsbeing detected?
ODC provides a structure for the defect data
21
Defect Causal Analysis
• Purpose: Characterize process issues that lead to injection of defects.
Step 1: ID a set of defects.
Step 2: Identify Defect Types with team of experts.
Step 3: Plot the distribution of Defect Types.
Step 4: Map Defect Types back to development activity.
Step 5: Develop action plan to address process deficiencies.
Step 6: Monitor process to ensure changes were effective.
22
Defect Type Mapped to Phase
H/L Level DesignInterface
Configuration MgmtBuild/Package/Merge
ImplementationTiming/Serialization
ImplementationAssignment
LLD/ImplementationChecking
Low Level DesignLogic
Architecture/HLDFunction
23
Sample Distribution
0%
20%
40%
60%
80%
100%
Inter
face
B/P/M
Logic
Checki
ng
Assign
men
t
Timin
g/Ser
ial
Functio
n
24
Defect Trigger Analysis
• Purpose: Characterize process issues that allowed defects to escape to later phases.
Step 1: ID a set of defects.
Step 2: Identify Defect Trigger with team of experts.
Step 3: Plot distributions: Defect Trigger by Family, Review Triggers, Function Test Triggers, System Test Triggers.
Step 4: Map Defect Trigger family back to detection activity(ies).
Step 5: Develop action plan to increase missed Defect Triggers.
Step 6: Monitor process to ensure changes were effective.
25
Defect Trigger Family Mapped to Phase
Testing in real-world scenarios or extreme scenarios
System Test
Testing of specified scenariosFunction Test
Peer Reviews or InspectionsReview/Inspection
26
Sample Distributions
Defect Triggers by Family
0
10
20
30
40
50
60
Review/Inspection Function Test System Test
Function Test Triggers
02468
1012141618
Seq
uenc
ing
Var
iatio
n
Inte
ract
ion
Cov
erag
e
Sim
ple
Pat
h
Com
plex
Pat
h
27
Additional Use for DTA
• Purpose: Determine quality status.
Step 1: Plot distribution of Defect Triggers after each detection activity.
Step 2: Compare to historical defect profile to determine if profile is as expected.
Step 3: Develop action plan, if necessary, to get back on track.
• If your project thinks it’s one phase, but the distribution is of an earlier phase, then you’re really in the earlier phase!
28
Agenda
• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary
29
The Big Picture: DP & ODC
• Mission: Improve software quality by using readily available data to decrease defects injected and increase defects detected.
Defect Prevention
RQTS DSGN CODE TEST Field
Ana
lysi
s Action
Ana
lysi
s Action
Ana
lysi
s Action
Ana
lysi
s Action
Ana
lysi
s Action
ODC Classification of Software Defects
30
Important Points about ODC
• A defect in the software is a defect in the process.• Implementing ODC is very cost-effective
– Enhances data already collected (software defects)– Adding fields that are completed real-time make data
collection virtually free!– Tooled to quickly identify process defects (mapping)
• ODC can be implemented in stages– Start with field defects, then move to in-process analysis– Utilize defect profiling in-process to predict quality and
project status
• Fields can be tailored to your own organization
31
References
• Handbook of Software Reliability Engineering (Michael R. Lyu, Editor; IEEE Computer Society Press/McGraw-Hill)
• Ram Chillarege (www.chillarege.com)• IBM Research Center for Software Engineering (
www.research.ibm.com/softeng)• Soheil Khajenoori, SIAGroup (
