Upload
isabelmargarido
View
85
Download
3
Tags:
Embed Size (px)
DESCRIPTION
What needs to be considered when doing process improvements in high maturity environments. An example of improving the efficiency of code reviews.
Citation preview
Portugal
Process Improvement in High Maturity
Environments
Improving the Efficiency of Code Reviews
André Ferreira
Software Quality Assurance Engineer
Critical Software S.A.
2013-10-18
2 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
Outline
1. SPI success - evaluation and measurement
2. CMMI High Maturity Processes Areas - a framework to drive
SPI initiatives
3. SPI initiative at Critical Software – Improving the efficiency of
Code Reviews
4. Recommendations/guidelines to improve the probability of
success of SPI initiatives.
3 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI success: reports on SPI Success.
Performance Results of CMMI-Based Process Improvement –Technical Report
CMU/SEI-2006-TR-004 ESC-TR-2006-004
Performance category
Medium Improvement
# Data points
Lowest Improvement
Highest Improvement
Cost 34% 29
3% 87%
Schedule 50% 22 2% 95%
Productivity 61% 20 11% 329%
Quality 48% 34 2% 132%
Customer Satisfaction
14% 7 -4% 55%
Return on Investment
[4.0:1] 22 [1.7:1] [27.7:1]
4 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI success: evaluation of SPI Success - I
Evaluation and validity of SPI success is questionable.
Hard to establish a clear relation of SPI initiative and SPI
results. Results are not questionable by themselves, but a clear
understanding of how success relates to SPI is unclear, mainly as result of:
1. Confounding factors – not considering relevant factors, validity threat
2. Incomplete context descriptions – context of SPI not fully described
3. Evaluation validity – threats to validity not discussed
4. Measurement validity – measurement definition inconsistencies
5. Measurement scope – missing the big picture
Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement -
A Systematic Literature Review. IEEE Transactions on Software Engineering
5 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI success: evaluation of SPI Success - I
Y(SPI sucess) = f ( f1, f2, f,3)
Causation
established
6 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI success: evaluation of SPI Success - I
Y(SPI sucess) = f ( f1, f2, f,3)
Causation
established
Presence of
confounding factors Y’(SPI sucess) = f’ (?, ?, f1, f2, f3, ?)
7 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI success: evaluation of SPI Success - II
Evaluation strategies for SPI initiatives:
Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement -
A Systematic Literature Review. IEEE Transactions on Software Engineeringl
Name Percentage of studies
Pre-post Comparison 49%
Statistical Analysis 15%
Pre-post Comparisons & Survey 7%
Statistical Process Control 5%
Coat benefit analysis 3%
Not identified 14%
8 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
CMMI High Maturity Framework Overview
Support
Pro
ce
ss
Ma
na
ge
me
nt
Pro
ject
Ma
na
ge
me
nt
CAR OPP
QPM
OPM BO
QPPOs PPBs
Defects
OIP
Corrective Actions Project QPPOs
Performance Issues
9 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative
Improving the Efficiency of
Code Reviews
10 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Opportunity Identification OPM.SP 1.1/OPM.SP1.2
1. Annual Review Business Objectives
Business objective: Improve organizational profit margins
QPPOs: Improve project profit margins – do the
same/better for less
2. Feedback from projects
Issues with review procedure – cumbersome - to much
time managing review records.
Lead to establish a performance baseline of total
appraisal cost for code reviews.
11 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Opportunity Identification OPM.SP1.2/OPM.SP 1.3
Analysis of performance data
Feedback from projects was confirmed to be an
issue – statistical analysis of a control chart.
Code review is a - fat process!
Opportunity identified:
Decrease appraisal cost of performing code
review procedure by:
Improving efficiency (less effort) but control
impact on effectiveness
12 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
(A) Appraisal and
(P) Preventive
costs
(I) Internal
and (E)
External
failure costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
SPI Initiative – Potential Improvement I OPM.SP 1.3
Cost of Quality (CoQ) Management model
13 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
(A) Appraisal and
(P) Preventive
costs
(I) Internal
and (E)
External
failure costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
SPI Initiative – Potential Improvement I OPM.SP 1.3
Cost of Quality (CoQ) Management model
14 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement I OPM.SP 1.3
BO
QPPOs
Processes
Pro
ce
ss M
an
ag
em
en
t
Project Gross
Margin >= Target
EBT
Rework Verification/ Code reviews
Rationale of impact on QPPOs
15 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement I OPM.SP 1.3
BO
QPPOs
Processes
Pro
ce
ss M
an
ag
em
en
t
Project Gross
Margin >= Target
EBT
Rework Verification/ Code reviews
Rationale of impact on QPPOs
Decrease on overall CoQ will
have a positive impact in the
ability of projects to meet
Project Gross Margin (QPPO)
16 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement II OPM.SP 1.3
Hypothesis1: Less effort spent performing code reviews will
contribute to a reduction in total appraisal cost.
Appraisal and
preventive
costs
Internal and
external failure
costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
17 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement II OPM.SP 1.3
Hypothesis1: Less effort spent performing code reviews will
contribute to a reduction in total appraisal cost.
Appraisal and
preventive
costs
Internal and
external failure
costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
18 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement II OPM.SP 1.3
Hypothesis1: Less effort spent performing code reviews will
contribute to a reduction in total appraisal cost.
Appraisal and
preventive
costs
Internal and
external failure
costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
19 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement II OPM.SP 1.3
Hypothesis1: Less effort spent performing each code review will
contribute to a reduction in total appraisal cost
Rationale of estimation of potential benefit
(1) Assuming a 10x decrease on managing
review records -> x hours saved per review
(2) y number of reviews performed each year
Total benefit (TH1)
TH1(hours) = x*y
TH1(€) = ~4K€/year
20 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement III OPM.SP 1.3
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (re-invest in Appraisal Cost)
Motivate an increase of # code reviews in projects with little
investment in code reviews
Motivate re-investment of effort saved to perform additional
code reviews (increase code coverage)
...with the final objective of decrease Total Cost of Quality by
decreasing considerably the Internal and External Failure
costs.
21 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement III OPM.SP 1.3
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (Increase Appraisal Costs)
Appraisal and
preventive
costs
Internal and
external failure
costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
22 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement III OPM.SP 1.3
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (Increase Appraisal Costs)
Appraisal and
preventive
costs
Internal and
external failure
costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
23 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement III OPM.SP 1.3
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (Increase Appraisal Costs)
Appraisal and
preventive
costs
Internal and
external failure
costs
Quality
Cost
Low
High
100%
good
100%
defective
Total cost of
quality
24 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Potential Improvement III OPM.SP 1.3
Estimation of potential benefit Based on performance baselines it takes 5x more to fix a
defect at ST when compared to code reviews
Projects who invest in code reviews, on average, have less
defects/LOC found in ST (corroborates the model) with a
typical distribution of [74%/24%] distribution of code reviews
VS. ST defects
Projects who invest less in code reviews have [5%/94%]
defect distribution
In the scenario of these projects with lower investment in code review
shifting to a bigger investment in code reviews, having an approximation
to a ratio of [74%/24/] we estimated potential savings of 70K€ for a year
timeframe.
TH2(€) = 70K€
25 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – ROI OPM.SP 1.3
Estimation of potential benefit
Hypothesis 1
Tool license cost: 3.6K
Management of SPI initiative:
Deployment + Training: 3K€
ROI [ 0.4:1]
Hypothesis 2
Tool license cost: 3.6K
Management of SPI initiative:
Deployment + Training: 3K€
ROI [ 9.6:1]
26 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Elicit Improvements OPM.SP2.1/SP2.2
A target was given to identify approaches that could
decrease in 10x the time to manage code review
records
Several tools were identified and analyzed in a DAR
initiative – ‘smoke test’.
Crucible came as the best option as it enabled to set-up
a review in less than 10 minutes among other relevant
factors related to infrastructure.
27 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Pilot OPM.SP2.3/SP2.4
Criteria of success in supporting H1.
Decrease review setup/manage records is
consistent below 10m/code review
Impact on defect detection ability (def/kloc) is
non negative
Pilot lasted 3 months and involved 5 project and a
workshop with SPA area members.
28 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Pilot Results OPM.SP2.4
Performance baseline with pilot results was created.
Review setup/manage records was under 10m
resulting in a 12x decrease on overall effort – change
in performance tested with statistical
significance.
Defect defection rate has not affected:
when compared to existing performance baseline
(44 defect/KLOC)
Overall, Crucible supported H1 and therefore a base to
approach H2 with a higher degree of confidence.
29 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Initiative – Deployment OPM.SP3.1/SP3.2/SP3.3
Phase currently in progress
– Evaluation of ROI will be over a 1 year period
– Monitoring cost of quality
– For the second year end beyond the NPV will
replace ROI for Cost/Benefit analysis of SPI
initiative
30 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
SPI Recommendations
Seek to establish your governance model (GM)
Use/create a GM that supports the rationale of your
network of Objectives vs. Processes
Start small, measure the least as possible, focus on
making the model reliable.
Only with a minimal statistical confidence on cause and
effect you are able to estimate ROI of SPI
Y(Repeatable SPI sucess) = f (f1,f2,f3)
Y’(Risky SPI initiative) = f‘(?, ?, f1, f2, ?)
31 © 2
013 C
MM
I P
ort
ug
al C
onfe
rence S
eri
es –
All R
ights
reserv
ed.
You can contact me at: [email protected]
or
Questions?