Measuring the Cost of Software Quality
Ingeniería para la Industria
g Q yof a Large Software Project
at Bombardier TransportationpPresented by Claude Y. Laporte, Eng., Ph.D.
ProfessorDepartment of Software and IT Engineeringp g g
École de technologie supérieure, Québec, Canada
Project Editor of ISO/IEC 29110 for Very Small Entities
Professor Normand Séguin, Ph.D.
Universitad de Lima, PeruSeptember 13th 2012September, 13th 2012
http://www2.ulima.edu.pe/
Ingeniería para la Industria
Content• IntroductionIntroduction
• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityand the Measurement of the Cost of Quality
• Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality
• Measurement of the Cost of Software Quality – A Case Study
• Conclusion
2
École de technologie supérieure
3
École de technologie supérieureOver 6,3000 studentsEach year 2,200 paid industrial internships in over 900 companies.
Students are paid about 36,500$ for 3 internships of 4 months
Undergraduate Programs• Software Engineering • 600 students (400 in Software Eng.)
20 P f i th d t t hg g
• IT Engineering• Construction Engineering• Production Engineering
• 20 Professors in the department have a mean industrial experience of 10 years.
• Electrical Engineering• Mechanical Engineering • Logistics and Operations Engineering
• Graduate Programs• Software Engineering• Information Technology
175 students• Information Technology• Other programs
www.etsmtl.ca4
Overview of my Background
National DefenseDefence Nationale
OerlikonOerlikonAerospaceAerospace
Defence Nationale
(1973‐1991)
(1991‐1999)
(1999‐2000)
f f d i i
5
Department of Software and IT Engineering
(2000 ‐ )
Project Editor of ISO/IEC 29110 Standards and Guides
f
Joint Committee for IT
Sub committee (SC) 7
Standardization of processes, supporting tools and supporting technologies for the Sub committee (SC) 7 technologies for theengineering of software products and systems.
Mandated to developWorking Group (WG) 24
Mandated to develop Standards and Guides for Very Small Entities *
6Very Small Entity - An entity (enterprise, organization, department or project) having up to 25 people.
ISO/IEC 29110 Standards and Guidesfor Very Small Entities (VSEs)for Very Small Entities (VSEs)
• Entry - Targets VSEs typically d l i 6 thdeveloping 6 person-monthprojects and start-up VSEs;
• Basic - Targets VSEs developing only one project at a time;
• Intermediate – Targets VSEs developing multiple projects
Advanced
Intermediate
p g p p jwithin the organizational context;
• Advanced – Targets VSEs which want to sustain and grow as an
Entry
Basicwant to sustain and grow as an independent competitive software development business.
(ISO/IEC 29110)
Very Small Entities are enterprises, projects or departments having up to 25 people.
7
Ingeniería para la Industria
Content
• IntroductionIntroduction
• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityand the Measurement of the Cost of Quality
• Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality
• Measurement of the Cost of Software Quality – A yCase Study
• Conclusion
8
Software Development – What the customer wants
9
Best Approach = Better Faster Cheaper
Why Software Fails ?Why Software Fails ?
‘Studies have shown that software specialists spend about 40% to 50% of their time on avoidable rework rather than on whatto 50% of their time on avoidable rework rather than on what they call value-added work, which is basically work that’s done right the first time.
Once a piece of software makes it into the field, the cost of fixing an error can be 100 times as high as it would have been g gduring the development stage.’
Software managers and software engineers ‘must’ select and use appropriate practices to reduce rework (i.e. waste or ‘scrap’)
Adapted from (Charette 2005) 10
Components of Project Cost
Project Cost
Cost of Quality Cost of PerformanceCost of Quality Cost of Performance
• Generation of plans• Software Development
Cost of NonConformance
Cost of Conformance
• Fixing defects• Re testing • Re‐reviews
Appraisal Costs
• Reviews• Inspections
Prevention Costs
• Training•Methodologies • Updating source code
• Modifying documents
Inspections• Testing• IV&V• Audits
•Methodologies• Tools• Data gathering
11
• Audits
Adapted from (Haley et al., 1995)
Cost of Quality
• Data from Professional Software Engineers Site A
AmericanSite A
AmericanSite B
EuropeanSite C
EuropeanSite D
EuropeanCourse A
2008Course B
2008Course C
2009Course D
2010Course E
2011Course F
2012AmericanEngineers
(19)
AmericanManagers
(5)
EuropeanEngineers
(13)
EuropeanEngineers
(14)
EuropeanEngineers
(9)
2008(8)
2008(14)
2009(11)
2010(8)
2011(15)
2012(10)
Cost of performance 41% 44% 34% 31% 34% 29% 43% 45% 45% 34% 40%
Cost of rework 30% 26% 23% 41% 34% 28% 29% 30% 25% 32% 31%
Cost of appraisal 18% 14% 32% 21% 26% 24% 18% 14% 20% 27% 20%pp
Cost of prevention 11% 16% 11% 8% 7% 14% 10% 11% 10% 8% 9%
Quality 71 8 23 35 17 403 19 48 35 60 55
Quality = Number of Defects/ 1,000 Lines of Code
12
Defect ‘Injection’ During DevelopmentD f t (%)Defects (%)
W id if d f iblWe must identify defects as soon as possible to reduce cost and schedule slippage.
Development PhasesAdapted from (Selby 2007) 13
Defect Detection During DevelopmentDefects Detected/Defects Detected/ Defects Injected
We must implement software engineering practicesWe must implement software engineering practicesto detect and correct 90% of defects as early aspossible to reduce cost and schedule slippage
Adapted from (Selby,2007)
Development Phases14
possible to reduce cost and schedule slippage.
Start of Initiative% f T l
Cost of Quality
40
45
Cost of Non Conformance (Rework)
Start of Initiative
41%% of TotalProject Cost
30
35
Appraisal & Prevention Costs
15
20
25
18%
5
10
15
11%
0
M t it L l
1 2 341988 1990
6%5%
Maturity Level 19901992
19961994
15Adapted from (Haley et al., 1995)
Ingeniería para la Industria
Content
• IntroductionIntroduction
• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityand the Measurement of the Cost of Quality
• Undergraduate/Graduate Software Quality Assurance Course and the Measurement of the Cost of Quality
• Measurement of the Cost of Software Quality – A yCase Study
• Conclusion
16
Software Quality Assurance Course at ETS• Fourteen Topicsp
– Chapter 1: Basic Software Quality Assurance Knowledge
– Chapter 2: Quality Culture and Cost of Quality (COQ)
Ch t 3 Q lit R i t
French Textbooks
– Chapter 3: Quality Requirements
– Chapter 4: Software Engineering Standards and Models
– Chapter 5: Reviews and COQ
Vol. 1
– Chapter 6: Software Audit
– Chapter 7: Verification and Validation and COQ
– Chapter 8: Tests and Software Quality AssuranceEnglish Textbook
p Q y
– Chapter 9: Software Configuration Management and COQ
– Chapter 10: Policies, Processes and Procedures and COQ
Chapter 11: Measurement and COQ
Vol. 2
– Chapter 11: Measurement and COQ
– Chapter 12: Management of Suppliers and Contracts
– Chapter 13: Risk Management and COQ
– Chapter 14: Software Quality Assurance Plan
17Publication in 2013
Software Quality Assurance CourseLaboratory
• Software Development Project with ISO/IEC 29110– Put in practice the software quality practices learned in class,
y
– Each team manages and executes a project using ISO/IEC 29110• A process is a component of the COQ (i.e. Prevention)
Statement S ft
CustomerImplementation
ProcessInitiation
Statement of Work
SoftwareConfiguration
Planning Control
Project Management Process
Analysis
Design
Measure the Cost of
Software
Organizational Management
Planning Control
Execution Closure
Construction
Tests
Delivery
Quality
18
Organizational Management Delivery
ISO/IEC 29110 Standard for Very Small Entities
Graduate Software Engineering Program
• Twelve 3-credit courses and a 9-credit Project in Software engineering
Software Engineering Program
engineering• Students are typically from industry with a minimum of 2 years
of experienceof experience– Students typically have an undergraduate degree in either computer
science or management information systems and work for small and di i d i timedium-sized organizations.
• Software Quality Assurance CourseStudents in teams of 3 have to do a Project in an– Students, in teams of 3, have to do a Project in an Organization
• Measurement of the Cost of QualityQ y
19
C t tIngeniería para la Industria
Content• Introduction
• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityy
• Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality
• Measurement of the Cost of Software Quality – A Case Study
• Conclusion
20
M i th C t f S ft Q litMeasuring the Cost of Software Quality of a Large Software Project
at Bombardier Transportation in Canada
Laporte C Y Berrhouma NLaporte, C. Y., Berrhouma, N., Doucet, M., Palza‐Vargas, E.
(Laporte et al. 2012) 21
Challenges Facing Railway Manufacturers
• Criticality of softwareBetter, Faster, Cheaper
y– Financially, environmentally or for
human safety.• Multi-disciplinary system development,• Integrator-Suppliers Relationships,• Multi-country development,• Multi-cultural teamsMulti cultural teams,• Downsizing/Merger/Turnover,• Off shoring.
ERTMS / ETCS (European Rail Traffic / ( pManagement System / European Train
Control System)
22
Overview of Bombardier• Workforce of some 80,000 people in over 24 countries.
• Bombardier AerospaceW ld l d i h f f b i j d i l– World leader in the manufacture of business jets and regional transport,
• Bombardier Transportation– Leader in the manufacture of rail transport equipment.,p q p
– Manufactures locomotives, freight cars, and propulsion and control systems,
Provides systems and signalling equipment– Provides systems and signalling equipment.
• Modern trains and subways are increasingly complex, and more and more subsystems are computer-controlled, such as propulsion and braking systems.
• At the time of this case study, there were over 30 software development centers within Bombardier Transportation for adevelopment centers within Bombardier Transportation, for a total of about 950 software engineers.
(Laporte et al. 2012) 23
The Software Development Group
f B b di T t ti i Q éb (C d )of Bombardier Transportation in Québec (Canada)• A group of 30 software engineers whose role is to design, develop,
and maintain embedded software for trains and subwaysand maintain embedded software for trains and subways• Software monitoring system used for collecting software
maintenance information, • Software for controlling car inclination.
SD
Media Converter
LINK PWR LINK
5VDC. 1A_ __ __ +
UP LINK
RX
TX
(Laporte et al. 2012) 24
Software Engineering Processof the Software Engineering Group *g g p
25* Located in Québec, Canada (Laporte et al. 2012)
ISO/IEC 15939 ‐ Systems and Software Engineering Measurement Process StandardEngineering ‐Measurement Process Standard
Technical &Management
Measurement Requirements User Feedbackg
Processes InformationProducts
InformationNeeds
Core Measurement Process
Establish Plan Perform EvaluateCommitment Measurement
E i B
Plan PerformanceInformation
Measurement Measurement
Experience Base
Improvement Actions
EvaluationResults
(ISO/IEC 15939)
Scope of ISO/IEC 15939
26
Objectives of the Measurement Projectf h S f D l Gof the Software Development Group
1. Identify a project to measure the Cost of Software Quality y p j Q y(CoSQ),
2. Collect data on costs, 3 Categorize costs related to software quality3. Categorize costs related to software quality,4. Develop a data model for the measurement of CoSQ,5. Analyze data collected, on the selected project, using the y , p j , g
data model,6. Present the CoSQ report to senior management,7 Expand the measurement of the CoSQ to other software7. Expand the measurement of the CoSQ to other software
projects in the Software Development Group.
(Laporte et al. 2012) 27
Overview of the Project SelectedOverview of the Project Selected
• Development of the software to control the subway of a large American city,
• Team of 15 software engineers,• L b f ft t k• Large number of software tasks
– 1,121 tasks• Total effort of the completed projectTotal effort of the completed project
– 88,598 hours
(Laporte et al. 2012) 28
The Five Stages of the Measurement Project/using the ISO/IEC 15939 Standard
1. Identification of the project tasks related to the CoSQ– Identify measurement requirements and resources
2. Development of a list of typical tasks related to the CoSQDefine data measures– Define data measures
3. Categorization of the tasks related to the CoSQ– Select and plan measures
4. Development and application of weighting rules– Define measurement criteria
5 D t i ti f th fid f th i hti l5. Determination of the confidence of the weighting rules– Define data collection and analysis
29(Laporte et al. 2012)
Stage 1. Identification of Project Tasks R l d h C SQRelated to the CoSQ
• Software engineering process is composed of 4 ‘types’Lif C l– Life Cycle,
– Process,
Type Process Element
- Sub-process, - Activity.
ypLife Cycle Primary Life Cycle ProcessProcess DevelopmentSub-process Code and DebugActivity Unit Tests
Representation of BSEP elements
WBS Element Task Name Effort [hours]
Code document Monitoring-Unit Testing 91.1
(Laporte et al. 2012)
Sample registration of a task in the Project accounting system
30
Stage 2. Development of a list of tasks l d h C f S f Q lirelated to the Cost of Software Quality
• A list of tasks related to the CoSQ is developed for this projectA list of tasks related to the CoSQ is developed for this project.
Primary Life Cycle ProcessesSupply
Organizational Life Cycle ProcessesManagementpp y
DevelopmentSystem Requirements AnalysisSystem Architectural DesignS ft R i t A l i
gEstimate the ProjectPlan the ProjectManage RisksT k th P j tSoftware Requirements Analysis
Software Architectural DesignSoftware Detailed DesignSoftware Coding and Testing
Track the ProjectInfrastructure
Plan Infrastructure ActivitiesEstablish the Project Infrastructure
Subset of CoSQ-related Tasks
Software IntegrationSoftware Validation Testing
Establish the Global InfrastructureMaintain the Infrastructure
(Laporte et al. 2012)
Q
31
Stage 3. Categorization of tasks l d h C SQrelated to the CoSQ
• All 1,121 tasks were sorted as follows: – Implementation (I), – Evaluation (E), – Prevention (P)– Prevention (P), – Rework (R) (internal and external anomalies).
TaskTaskIdentification WBS Element Task Name I E P R
2410 Code Trace Requirements - - x -
Example of classification of the Requirements Traceability Task
(Laporte et al. 2012) 32
Stage 4. Development of weighting rules• Many tasks belong to more than one CoSQ category
– e.g. The ‘Test and coding’ task overlaps the Evaluation and Implementation CoSQ categories. p Q g
• Twenty-seven (27) weighting rules have been defined. Rule
Number Name of Rule Typical Tasks Implementation Evaluation Prevention ReworkNumber
3 Process Audit Software Project & Process Audit 100%
7 Review Design review 100%7 ev ew es g ev ew 00%
12Problem Correction and Coding
Corrections, Debugging and Final Coding
30% 70%
17 Training Training of new resources 100%
26 Follow-up & Follow-up & Validation (all 85% 15%26 Validation Validation (all releases)
85% 15%
(Laporte et al. 2012) 33
Stage 5. Determining the Confidence f W i h i R lof Weighting Rules
• What is the level of confidence of the CoSQ measures ? • Another component was added to the rules
– "H" for high precision,– "M" for medium precision and– M for medium precision, and– "L" for low precision.
C fid ith th W i hti R l P t (%)Confidence with the Weighting Rules Percentage (%)
H High 88%
M Medium 11%
L Low 0,2%
Total number of activities 1,121
(Laporte et al. 2012) 34
Cost of Software Quality Measurement Results
Perform Evaluation Prevention Rework CoSQQ
Hours 59,231 18,551 2,025 8,791 28,824
P t 67 % 17 % 4 % 11 % 32 %Percentage 67 % 17 % 4 % 11 % 32 %
• Total effort (i e cost) of the project: 88 598 ho rs• Total effort (i.e. cost) of the project: 88,598 hours• Number of software tasks: 1,121 tasks
(Laporte et al. 2012) 35
Cost of Software Quality at Raytheon compared to Bombardier Transportationp p
70 CMM level 3Start of intiative CMM level 1
50
60
ct c
ost TCoSQ
Rework
Total Cost of Software Quality (40%) 32 %
40
of to
tal p
roje
c
Cost ofConformance Percentage of Rework ( ≈13%)
Percentage of Evaluation ( ≈20%) 17 %
11 %
20
30
Per
cent
age
o Conformance Percentage of Rework ( ≈13%)
Percentage of P ti ( 7%) 4 %
11 %
0
10
P
PreventionRework
AppraisalPrevention ( ≈7%) 4 %
0
Year87 88 89 90 91 92 93 94 95 96
36Adapted from (Haley 1996)
Cost of Quality and Maturity Levels
• A study of the Software Engineering Institute (SEI) showed that rework varies between 15% and 25% of the cost of
CMM M t it L l P t f R k
that rework varies between 15% and 25% of the cost of developing a CMM Maturity Level of 3.
CMM Maturity Level Percentage of Rework
1 ≥ 50 %2 25 % to 50 %3 15 % to 25 %4 5 % to 15 %5 ≤ 5 %
11 %
Relationship between CMM Maturity Levels and percentage of rework
(Gibson et al. 2006) 37
Recommandations1. Continue data collection and CoSQ Measurement
– Continue the data collection using procedures and methods (for example use of tools database structures) within the SDGexample use of tools, database structures) within the SDG.
2. Control activities related to the correction of problems– Most defects are found mainly during the execution of software
i f d i d di krequirements, software design and coding tasks• Reduce the defects by increasing prevention effort
– e.g. Execute more peer reviews (walk-through, inspection)
3. Present the CoSQ measurement results to Management– To enable them to develop better budgets for business process
improvementimprovement. – Establish a scoreboard to display the CoSQ and the results of
process improvement activities to illustrate the relationships between thembetween them.
38(Laporte et al. 2012)
Recommandations4. Measure the CoSQ at other Bombardier Transportation sites
– To identify opportunities for reducing costs,T id b i b d d l d li i i i– To provide a basis to budget development and quality activities,
– To use the results of CoSQ measures to improve processes.
OITasks
OutputsInputs
Project WBSCoSQ Report
1. Identify CoSQ tasks, using Project Plan and WBS
2. Record CoSQ effort by category
3. Apply weighting rules CoSQ Measurement Process
Typical CoSQ list of tasks
Weighting rules
Findings
Recommendations
pp y g g
4. Compute the CoSQ
5. Develop findings and recommendations
6. Review findings and recommendations with management
Entry Criteria Exit Criteria
management
7. Produce final CoSQ Project Report
8. Archive Report
39(Laporte et al. 2012)
Entry CriteriaMeasures
Exit Criteria
Project Plan Approved CoSQ Project Report Approved
Effort (staff‐hours)
C t tIngeniería para la Industria
Content
• IntroductionIntroduction
• Business Rationale for Software Quality Assurance and the Measurement of the Cost of Qualityand the Measurement of the Cost of Quality
• Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality
• Measurement of the Cost of Software Quality – A yCase Study
• Conclusion
40
ConclusionConclusion• The concept of Cost of Software Quality is a powerful
tool to help quantify the rework indentify weaknessestool to help quantify the rework, indentify weaknesses and prevention activities,
• The Software Quality Assurance courses we teach help Q y psoftware engineers in understanding the value of the quality assurance practices learned in class and put into
ti i th l b t ipractice in the laboratories,• The Case study at Bombardier Transportation
demonstrated the business value of the measurement ofdemonstrated the business value of the measurement of the Cost of Software Quality.
Best Approach Better Faster Cheaper
41
Best Approach = Better Faster Cheaper
Ingeniería para la Industria
Th k f iThank you for your attention
42
Contact Information
• Claude Y Laporte– Voice: + 1 514 396 8956Voice: + 1 514 396 8956– E-Mail: [email protected]– Web: http://profs.etsmtl.ca/claporte/English/index.html
• Public site of WG 24l k i i l d– Free access to Deployment Packages, presentation material and
articles:• http://profs.logti.etsmtl.ca/claporte/English/VSE/index.htmlp p g p g
43
References• Fanmuy, G., L’Ingénierie et Management des Systèmes pour les PME/TPE et petits projets, Association Française d'Ingénierie Système
(AFIS)/International Council on Systems Engineering (INCOSE) May 24th 2011 Paris France(AFIS)/International Council on Systems Engineering (INCOSE), May 24th, 2011, Paris, France.• Gibson, D.L., D.R. Goldenson, and K. Kost, Performance Results of CMMI®-Based Process Improvement. Technical Report, CMU/SEI-2006-
TR-004, ESC-TR-2006-004, 2006• Haley, T., Ireland, B., Wojtaszek, E., Nash, D., Dion, R., Raytheon Electronic Systems Experience in Software Process Improvement, Technical
Report CMU/SEI-95-TR-017, Software Engineering Institute, November 1995.• Iberle K., « But Will It Work For Me », Proceedings of the Pacific Northwest Software Quality Conference, p. 377-398, Portland, United States , , g Q y , p , ,
2002,• Iberle K., « They Don’t Care About Quality », Proceedings of STAR East, Orlando, United States, 2003.• ISO/IEC JTC1/SC7 N3288, New Work Item Proposal – Software Life Cycles for Very Small Enterprises, Mai 2005.• ISO/IEC 12207:2008, Information technology – Software life cycle processes, International Organization for Standardization/ International
Electrotechnical Commission: Geneva, Switzerland. • ISO/IEC 29110 - Lifecycle Profiles for Very Small Entities (VSEs) – Part 1: Overview. International Organization for
Standardization/International Electrotechnical Commission: Geneva, Switzerland.• ISO/IEC 15289:2006 - Systems and software engineering - Content of systems and software life cycle process information products
(Documentation)• Kabli, S., Conception, réalisation et mise a l’essai de trousses de déploiement pour faciliter et accélérer l’implémentation de la norme ISO/CEI
20000 l è i ÉTS 200920000 par les très petites structures, ÉTS, 2009.• Konrad, M., Overview of CMMI Model, Presentation to the Montréal SPIN, November 21, 2000, Montréal, Canada.• Kroll, P.; Kruchten, P.; The Rational Unified Process Made Easy – A Practionner’s Guide to the RUP.; Addison-Wesley, 2003 • Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software
Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p 14-31L t Cl d Y F G thi Pt k K Th D l t f S t E i i I t ti l St d d d S t T l f V• Laporte, Claude, Y, Fanmuy, Gauthier, Ptack, Ken, The Development of Systems Engineering International Standards and Support Tools for Very Small Enterprises, 22nd Annual International Symposium of the International Council on Systems Engineering, Rome, July 9-12, 2012.
• Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p 14-31.
• Laporte, C.Y., Alexandre, S., O’Connor, R., A Software Engineering Lifecycle Standard for Very Small Enterprises, in R.V. O’Connor et al. (Eds.): EuroSPI 2008, CCIS 16, pp. 129–141.(Eds.): EuroSPI 2008, CCIS 16, pp. 129 141.
• Long, L., The Critical Need for Software Engineering Education, Crosstalk - The Journal of Defense Software Engineering, January 2008, pp 6-10.
• Reifer D., « Let the Numbers Do the Talking », Crosstalk – The Journal of Defense Software Engineering, march 2002.• Rogers, Everett M., Diffusion of Innovations, fifth edition, Free Press, New York, 2003. 44