1
Pragmatic Prioritization of Software Quality Assurance Efforts
Emad ShihabQueen’s University
2
Motivation
Software ^ has bugsalways
always
How to allocate quality assurance efforts for legacy and new code?
Q:
managers ^ have limited resources
3
One Answer: Defect Prediction
Many recent studies focused on defect prediction
….However adoption in practice remains low
4
An Industrial Example
How to prioritize the creation of unit tests for a large legacy system
5
Criteria for Pragmatic QA Approaches
Focused Granularity
Unit of prioritization must be focused (e.g., function or method)
Current Approach Desired Approach
File, Folder or subsystem level (Thousands of LOC)
6
Criteria for Pragmatic QA Approaches
Responsive Feedback
Continuous reflection of system changes.
Current Approach Desired Approach
Predict for next release
7
Criteria for Pragmatic QA Approaches
Consider Effort
Consider effort in recommendations (e.g., use size)
Current Approach Desired Approach
No consideration for effort
In addition, need to evaluate the generality of the approaches to understand implications of findings
8
Proposed Pragmatic SQA Approaches
Unit test creation
How to allocate quality assurance efforts for legacy code?Q:
A:
Change risk
How to allocate quality assurance efforts for new code?
Q:
A:
Complete
In Progress
9
Unit Test CreationApproach
1. Search modification record comments for keywords, bug identifiers and modified files
2. Extract source code of modified file(s) and compare to previous version to identify changed functions
3. Combine data from 1 and 2 to calculate function-level heuristics
10
Unit Test CreationHeuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
11
Unit Test CreationHeuristic Performance
Based on a heuristic, generate list of 10 functions to write unit tests for
Use size of function to measure effort required to write unit test
12
Unit Test Creation Usefulness: Was writing the unit test useful?
Time to write unit testA
B
C
6 bug fixes
2 bug fixes
0 bug fixes
Usefulness = 2/3 = 66.67%
13
Unit Test CreationUsefulness of Heuristics
Random
Most Recently Modified (MRM)
Size Risk (SR)
Change Risk (CR)
Most Recently Fixed (MRF)
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Modified (LM)
Largest Fixed (LF)
0 10 20 30 40 50 60 70 80 90 100
27.7
43.1
48.8
55
56.9
80
83.8
84.7
87
8.5
9.9
12.6
17.4
16
28.1
32.3
32.9
44.7
Open SourceCommercial
Revisiting the Pragmatic QA CriteriaUnit Test Creation
Unit Test CreationFunction level recommendations
Heuristic lists are continuously updated
Size of function used to consider effort required
Focused Granularity
Consider Effort
Responsive Feedback
Evaluate Generality: Studied Open Source and Commercial systems
15
Change RiskProposed Approach
1. Mine changes and extract change metrics
2. Identify bug-inducing changes
3. Predict risky changes
Revisiting the Pragmatic QA CriteriaChange Risk
Change RiskMap changes to modified functions
Flag risky changes while they are fresh in developer’s minds
Size of change used to consider effort
Focused Granularity
Consider Effort
Responsive Feedback
Evaluate Generality: Study on Open Source and Commercial systems
Thank youI am grateful for this opportunity!
Backups
Revisiting the Pragmatic SQA Criteria
Unit Test Creation Change Risk
Focused Granularity
Estimate Effort
Evaluate Generality
Timely Feedback
Pragmatic SQA Criteria
What we have What we wantFile level Unit of prioritization
must be focused (e.g., function)
Act on results in future release
Must be able to act on results of approach quickly
No effort provided Provide an estimate of the effort required for the task
Focused Granularity
Estimate Effort
Timely Feedback
Evaluate Generality
To help understand implications of findings
21
Changes RiskNew code
Identify risky changes for closer examination before they are integrated into the code base
22
Unit Test CreationImplemented code
Prioritize the creation of unit tests for implemented code
23
Revisiting the Pragmatic-aware Criteria
Focused Granularity Timely feedback Estimate Effort Evaluate Generality
24
Avoid the most bugs effectively!
Write unit tests for functions with best Return on Investment (ROI)
How can we avoid the most bugs given limited resources?
25
Defect Prediction... An Example
Uses code metrics and pre-release defects to predict files with post-release defects in the Eclipse project.
26
Motivation
Software has^ bugs and
managers have ^ limited resources
always
always
How to allocate quality assurance efforts for implemented and new code?
Q:
A: Re-active and pro-active SQA
27
Proposed ApproachExtracting Historical Data
1. Extract change data to calculate metrics (e.g., churn)2. Use SZZ to identify and predict bug-inducing changes3.
28
Answer … Defect Prediction!
Use code metrics and process metrics to predict which files are most likely to contain defects in future releases
29
Defect Prediction... The Response
“The results are interesting, but of limited use to us ...”
... Why?
30
Defect Prediction... Lessons Learned
1. Identifying files is too coarse of a granularity
2. Having to wait till the next release is too late
3. Need an estimate of effort required
4. Why should we believe this will work for us?
31
Criteria for Pragmatic SQA Approaches
Timely Feedback
Must be able to act on results of approach quickly
32
Criteria for Pragmatic SQA Approaches
Estimate Effort
Provide an estimate of the effort required for the task
33
Criteria for Pragmatic SQA Approaches
Evaluate Generality
Evaluate on multiple projects/domains to clearly understand implications of the findings