Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
TMMi history benefit and future
Klaus Olsen CEO of TMMi Foundation
klausolsentmmiorg
Session 3
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
Klaus Olsen
Author of ldquoSoftwaretest ndash how to get startedrdquo in Danish Founding member of TMMi reg Foundation CEO of the TMMi Management Executives Member of ISTQB representing Denmark Co-author of ISTQB Foundation and Advanced Syllabus Chair of ISTQB Foundation Working Group Certified ISTQB Foundation and Test Manager Advanced Certified Scrum Master
Founder and owner of the company Softwaretestdk since 2000
Has used the past 23 years to focus on software testing test process improvements teaching and mentoring
EuroSTARacute98 in Muumlnich Germany Second World Congress on Software Quality 2000 in Yokohama Japan EuroSTARacute2001 in Stockholm Sweden Quality Week 2001 in San Francisco USA EuroSTARacute2003 in Amsterdam Holland ASTA 2007 in Seoul Korea Test 2008 in New Delhi India EuroSTARacute2008 in Haag Holland ANZTB Test2009 conference Sydney Australia JSTQB カンファレンス 2010 in Tokyo Japan ASTA 2010 in Seoul Korea Czechtest 2011 in Prague Czech Republic TAPOST 2011 in Riga Latvia TMMi World Conference 2011 in Seoul Korea Czechtest 2012 in Prag Czech Republic Nordic Testing Days 2012 in Tallinn Estonia ANZTB Test2013 conference in Canberra Australia FiSTB Testing Assembly 2013 in Helsinki Finland Testing Portugal 2013 in Lisbon Portugal DSTB 2014 in Copenhagen Denmark Sixth World Congress on Software Quality 2014 in London UK Teszt amp Tea 2014 in Budapest Hungary Czechtest 2015 in Prag Czech Republic SEETEST 2015 Sofia Bulgaria
Public presentation includes
Experience with TMM - TMMi
First experience with TMM in 1997 and presentation about this at EuroSTARacute98 in Munich Germany
TMM is developed by Illinois Institute of Technology and published in two articles in US Air Force magazine Crosstalk in 1996
Meeting with Ilene Burnstein author of TMM in USA in 2000
Ilene Burnstein has described her original TMM model in detail in her book
Several assessment of companies in Denmark and Norway during the past 10 years
Participating in creating the current TMMi model
Those who cannot remember the past are condemned to
repeat itGeorge Santayana1863 in Spain ndash 1952 in Italy
He was a philosopher essayist poet and novelist
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
Klaus Olsen
Author of ldquoSoftwaretest ndash how to get startedrdquo in Danish Founding member of TMMi reg Foundation CEO of the TMMi Management Executives Member of ISTQB representing Denmark Co-author of ISTQB Foundation and Advanced Syllabus Chair of ISTQB Foundation Working Group Certified ISTQB Foundation and Test Manager Advanced Certified Scrum Master
Founder and owner of the company Softwaretestdk since 2000
Has used the past 23 years to focus on software testing test process improvements teaching and mentoring
EuroSTARacute98 in Muumlnich Germany Second World Congress on Software Quality 2000 in Yokohama Japan EuroSTARacute2001 in Stockholm Sweden Quality Week 2001 in San Francisco USA EuroSTARacute2003 in Amsterdam Holland ASTA 2007 in Seoul Korea Test 2008 in New Delhi India EuroSTARacute2008 in Haag Holland ANZTB Test2009 conference Sydney Australia JSTQB カンファレンス 2010 in Tokyo Japan ASTA 2010 in Seoul Korea Czechtest 2011 in Prague Czech Republic TAPOST 2011 in Riga Latvia TMMi World Conference 2011 in Seoul Korea Czechtest 2012 in Prag Czech Republic Nordic Testing Days 2012 in Tallinn Estonia ANZTB Test2013 conference in Canberra Australia FiSTB Testing Assembly 2013 in Helsinki Finland Testing Portugal 2013 in Lisbon Portugal DSTB 2014 in Copenhagen Denmark Sixth World Congress on Software Quality 2014 in London UK Teszt amp Tea 2014 in Budapest Hungary Czechtest 2015 in Prag Czech Republic SEETEST 2015 Sofia Bulgaria
Public presentation includes
Experience with TMM - TMMi
First experience with TMM in 1997 and presentation about this at EuroSTARacute98 in Munich Germany
TMM is developed by Illinois Institute of Technology and published in two articles in US Air Force magazine Crosstalk in 1996
Meeting with Ilene Burnstein author of TMM in USA in 2000
Ilene Burnstein has described her original TMM model in detail in her book
Several assessment of companies in Denmark and Norway during the past 10 years
Participating in creating the current TMMi model
Those who cannot remember the past are condemned to
repeat itGeorge Santayana1863 in Spain ndash 1952 in Italy
He was a philosopher essayist poet and novelist
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Klaus Olsen
Author of ldquoSoftwaretest ndash how to get startedrdquo in Danish Founding member of TMMi reg Foundation CEO of the TMMi Management Executives Member of ISTQB representing Denmark Co-author of ISTQB Foundation and Advanced Syllabus Chair of ISTQB Foundation Working Group Certified ISTQB Foundation and Test Manager Advanced Certified Scrum Master
Founder and owner of the company Softwaretestdk since 2000
Has used the past 23 years to focus on software testing test process improvements teaching and mentoring
EuroSTARacute98 in Muumlnich Germany Second World Congress on Software Quality 2000 in Yokohama Japan EuroSTARacute2001 in Stockholm Sweden Quality Week 2001 in San Francisco USA EuroSTARacute2003 in Amsterdam Holland ASTA 2007 in Seoul Korea Test 2008 in New Delhi India EuroSTARacute2008 in Haag Holland ANZTB Test2009 conference Sydney Australia JSTQB カンファレンス 2010 in Tokyo Japan ASTA 2010 in Seoul Korea Czechtest 2011 in Prague Czech Republic TAPOST 2011 in Riga Latvia TMMi World Conference 2011 in Seoul Korea Czechtest 2012 in Prag Czech Republic Nordic Testing Days 2012 in Tallinn Estonia ANZTB Test2013 conference in Canberra Australia FiSTB Testing Assembly 2013 in Helsinki Finland Testing Portugal 2013 in Lisbon Portugal DSTB 2014 in Copenhagen Denmark Sixth World Congress on Software Quality 2014 in London UK Teszt amp Tea 2014 in Budapest Hungary Czechtest 2015 in Prag Czech Republic SEETEST 2015 Sofia Bulgaria
Public presentation includes
Experience with TMM - TMMi
First experience with TMM in 1997 and presentation about this at EuroSTARacute98 in Munich Germany
TMM is developed by Illinois Institute of Technology and published in two articles in US Air Force magazine Crosstalk in 1996
Meeting with Ilene Burnstein author of TMM in USA in 2000
Ilene Burnstein has described her original TMM model in detail in her book
Several assessment of companies in Denmark and Norway during the past 10 years
Participating in creating the current TMMi model
Those who cannot remember the past are condemned to
repeat itGeorge Santayana1863 in Spain ndash 1952 in Italy
He was a philosopher essayist poet and novelist
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
EuroSTARacute98 in Muumlnich Germany Second World Congress on Software Quality 2000 in Yokohama Japan EuroSTARacute2001 in Stockholm Sweden Quality Week 2001 in San Francisco USA EuroSTARacute2003 in Amsterdam Holland ASTA 2007 in Seoul Korea Test 2008 in New Delhi India EuroSTARacute2008 in Haag Holland ANZTB Test2009 conference Sydney Australia JSTQB カンファレンス 2010 in Tokyo Japan ASTA 2010 in Seoul Korea Czechtest 2011 in Prague Czech Republic TAPOST 2011 in Riga Latvia TMMi World Conference 2011 in Seoul Korea Czechtest 2012 in Prag Czech Republic Nordic Testing Days 2012 in Tallinn Estonia ANZTB Test2013 conference in Canberra Australia FiSTB Testing Assembly 2013 in Helsinki Finland Testing Portugal 2013 in Lisbon Portugal DSTB 2014 in Copenhagen Denmark Sixth World Congress on Software Quality 2014 in London UK Teszt amp Tea 2014 in Budapest Hungary Czechtest 2015 in Prag Czech Republic SEETEST 2015 Sofia Bulgaria
Public presentation includes
Experience with TMM - TMMi
First experience with TMM in 1997 and presentation about this at EuroSTARacute98 in Munich Germany
TMM is developed by Illinois Institute of Technology and published in two articles in US Air Force magazine Crosstalk in 1996
Meeting with Ilene Burnstein author of TMM in USA in 2000
Ilene Burnstein has described her original TMM model in detail in her book
Several assessment of companies in Denmark and Norway during the past 10 years
Participating in creating the current TMMi model
Those who cannot remember the past are condemned to
repeat itGeorge Santayana1863 in Spain ndash 1952 in Italy
He was a philosopher essayist poet and novelist
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Experience with TMM - TMMi
First experience with TMM in 1997 and presentation about this at EuroSTARacute98 in Munich Germany
TMM is developed by Illinois Institute of Technology and published in two articles in US Air Force magazine Crosstalk in 1996
Meeting with Ilene Burnstein author of TMM in USA in 2000
Ilene Burnstein has described her original TMM model in detail in her book
Several assessment of companies in Denmark and Norway during the past 10 years
Participating in creating the current TMMi model
Those who cannot remember the past are condemned to
repeat itGeorge Santayana1863 in Spain ndash 1952 in Italy
He was a philosopher essayist poet and novelist
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Those who cannot remember the past are condemned to
repeat itGeorge Santayana1863 in Spain ndash 1952 in Italy
He was a philosopher essayist poet and novelist
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Crosby1979
CMM v10 1991
TMM1996
TPI 1997
CMM v111993
TMMDK version
2000
CMMi2000
TMMi reg
2005
TMMi reg
full version
2010
Maturity History Timeline
TMMi Foundation established2005 in London UK
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Foundation - Board of Directors
Chairman
9
Les Murray Fran OrsquoHaraMac Miller
Vice-chair
Geoff Thompson
Treasure
Secretary CEO
Klaus Olsen
Elected by the member of TMMi Foundation
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Foundation - Board of Advisor
10
Lev Lesokhin Brian FordBill Curtis
Steve Connell Brian Wells
Appointed by the BoD
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Foundation ndash Management Executives
CEO
11
Marc Gebauer
Erik van Veenendaal
Deputy CEO
Meile Posthuma
Accreditation
Klaus Olsen Andrew Goslin
Secretary Marketing
Geoff Thompson
TMMi Professional
Training
Appointed by the BoD
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Foundation is
33 Certified companies among them 5 from Korea TMMi Level 2
ITSTAR Co LTD (Development amp Operation Division)
Onycom Inc (QA ICT Solution IT Service Division)
TMMi Level 3 LG CNS (DC Test and Performance Test Teams)
Hanwha Thales Co LTD (RampD Division)
LIG Nex1 (Research amp Development Division)
22 training providers among them STA from Korea
15 supporting organisation
1757 LinkedIn members
1686 members of the TMMi Foundation
12
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Test Maturity Model Integration TMMi reg
Level 1 Initial
Level 5 Optimization Defect Prevention
Test Process Optimization
Quality Control
Level 3 Defined Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Level 4 Measured Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Level 2 Managed Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
TMMi reg maturity levels and process areas
16 Process Areas
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi reg Structure and components
Maturity Levels
Process Areas
Generic andSpecific Goals
Generic andSpecificPractices
Indicates Testing Process Capability
Achieves Goals
contains
contains
contains Describes Implementation
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi reg Structure and components
Maturity Level
Process Area 1 Process Area 2 Process Area n
Specific GoalsGeneric Goals
SpecificPractices
GenericPractices
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi reg Structure and components
Managed
Test Policy and Strategy
Test Planning Process Area
Develop a test plan
Institutionalize aManaged Process
Establish the test plan
Plan theprocess
TMMi level 2
SpecificGoal 4
SpecificPractice 45
GenericGoal 2
GenericPractice 22
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Foundation benefit we provide
TMMi model free available from our website in pdf
TMMi Professional CertificationTMMi Professional Syllabus from our website in pdf
TMMi Professional trial exam questions from our website in pdf
TAMAR requirement to build an assessment method
TAM TMMi Foundation provide an assessment method on license
18
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
How you can utilise TMMi reg
If you donacutet know where you are
a map wonacutet helpWatts Humphrey
If you donacutet know where you want to go then any road will do
TMMi reg provides the framework for anassessment of your companyacutes now situation
TMMi reg provides a map of improvement
possibilities from level 2 to level 5
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 Start by using the TMMi reg modelto locate your GPS positionon the TMMi reg Model
2 Once you know whereyou are use the TMMi reg
model as a reference for where you want to go
How you can utilise TMMi reg
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
How you can utilise TMMi reg
Study by measuring your companiestest maturity level through an assessment
Act by identifying the weakest areas in your company when using the TMMi reg Model
Plan by creating a test improvement plan based on the TMMi reg Model
Do by implementing the identified task from Act and Plan above
Plan
Do
Study
Act
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Assessment ndash how they are donea story
Engaging with a company who needs
an assessment
Should represent both large and
small size projects
Should represent the different
development methods used
Selecting 3-5 projects in the company
Project Manger
Test Manger
Developer
Tester
Select 3-4 people from each project for
interview
Often also 1 or 2 persons from a
Qualtiy or Method department
X questions for each process areaRun interview with all participants
Test strategy
Test Plan
Test case examples
Test reports
And more
Get access to artefacts discussed in
interviews
Score result for each project for
all process areas
Evaluate all input from interviews and
artefacts
Create overall TMMi assessment
report with result
Create proposal for an Action Plan
based on result of assessment
Highlight good practices they
must repeat
Go through proposal action plan
Highlight areas in need of an
improvementPresent result of assessment for the
company
Let the company decide what to do
next
AssessmentStorytelling
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
bull 96 DDP reached in 2004
bull Better and more stable quality
bull Moved from 78 DDP to 96 DDP during 3 years
Test effectiveness = Numbe
r of Defects found during t
est total number of defect
s after 90 days
96 defect detection rate reached
Effectiveness of test in company
Time for test Effectiveness of test
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
96 DDP reached
Better and more stable quality
Moved from 78 to 96 DDP during 3 years
Test effectiveness = Number of Defects found during test total number of defects after 90 days
96 defect detection rate
Year Test Effectiveness in Average Time for Testof total project
200x 78 DDP 23200x+1 94 DDP 28200x+2 96 DDP 25200x+3 95 DDP 28200x+4 92 DDP 19
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
bull Improved higher level of quality in projects
bull Now some with a 100 DDP others between 75 and 90DDP in average DDP were 92
Metrics from a company using TMMi
Effectiveness in projects
Time for test Effectiveness of test
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 TMMi Foundation History
2 Benefit of using TMMi Model
3 The future of TMMi Foundation
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi our future
TMMi and Agile
Alumni Club Free for all TMMi Professional Certified
Able to vote on our AGM for the BoD
Able to participate in our working groups
28
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Currently TMMi reg Foundation are working with the following organizations to grow recognition of the TMMi reg Model on a global scale
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Professional Training providers
30
Belgium Bolivia China Denmark France Germany
Hungary India India India India Ireland
Ireland Korea Malaysia Netherlands Netherlands Scotland
Spain UK UK UK USA
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Make a Difference and use TMMiClive Bates clivebatesexperimentuscom
Session 4
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 The challenges you face
2 Examples of client benefits
3 How to move forward with TMMi
4 TMMi trends
5 Donrsquot fear failure ndash embrace it
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Experimentus services
3
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Typical reasons for test improvement
Many Organizations are still facing tough business objectives in the current climate eg
bull Reduced time to market
bull Increased functionality and complexity
bull Increased reliability
bull Increased speed of performance
bull Increased number of users
bull Reduced prices being paid (cheaper productsapplications)
bull Need to invest in and utilizing new technologies and compete on a global stage Need to innovate or be left behind
bull Re-design of the workplace including co-developed or outsourced development
bull Meeting client requirements
Therefore
It can be said that applications are now more challenging to build
4
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
The result of all this is helliphellip
bull Systems and applications are playing an ever increasing part in every day life (social and business)
bull There is a greater emphasis to focus on quality aspects
bull Customers and users are becoming more and more demanding
bull Software failures are much more visible now and can have major impact on day to day activities ndash economically socially and internationally etc
bull Testing is becoming a key activity for helping to improve quality and the performance of the development process
bull The fast pace of change in the industry ndash consider what has happened over the last 5 or 10 years What will we see next
5
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Whose having assessments amp where
bull Service providers
bull Mining company
bull Pharmaceutical
bull Banking
bull Financial Services
bull Retail
bull Insurance
bull Mobile Telephone
bull Consulting System Integrators Outsourcing
bull Utilities
bull Software developers
bull TV and Communications
bull Etchellip
6
bull UK
bull India
bull USA
bull Malaysia
bull Australia
bull China
bull Ukraine
bull Spain
bull France
bull Mexico
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Example benefits seen - summary
bull Retain andor gain contracts Customers increasingly want their suppliers to be certified to a specific TMMi level if they want to have the business
bull Releases now meeting scheduled dates
bull Organisation have working processes in place for projects ndashno longer a game of chess consistent approach
bull Management receiving relevant progress information through correct metrics
bull ldquoFerrarirdquo type processes not ldquobicyclerdquo ones
bull Conducted internal assessments over period of time and introduced relevant improvements ndash achieved TMMi levels straight away
7
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Example benefits seen - summary
bull Able to involve product owners and identified risks were handled much better
bull Able to motivate staff
bull Found shortcomings in risk and processes despite having had an industry based audit that did not find problems
bull Improved quality products smoother releases
bull We can help quantify savings through improvement
8
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
How to improve
bull Start simple ndash do an online checkout first bull httpwwwexperimentuscomcustomer-journeyassess
bull Use the TMMi Framework as a guide
bull Be realistic what target level you want to aim for
bull Have an informal assessment to your target TMMi level ndash use the framework against your organisation and type of business
bull Identify the strengths and any gaps
bull Hold a workshop to discuss the gaps
bull Create a strategy defining what is to be improved and when and how they will be monitored
9
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
How to improve
bull Get yourself a championowner
bull Communicate what you are doing and why
bull Split improvements to quick wins and longer term goals
bull Have access to a TMMi expert to help you
bull Allocate the right amount of time to do this
bull Gain commitment from individuals and ensure they are open to change
bull Have an improvement plan and target metrics
bull Use an improvement model like IDEAL
COMMUNICATE PROGRESS
10
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
IDEAL model
11
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Definemonitor improvements
12
Priority Item Specific Goals Specific Practices Improvement Opportunity Task to Address Activity Individuals Involved Individual LeaderEstimated Effort
( Days)
Target
Completion DateDependencies
Training Mentoring
RequirementsStatus
Need Additional
ClarificationComments Questions
21 Test Policy amp
Strategy
SG1 Establish a Test
Policy
1SP 11 Define Test Goals Define and maintain test goals based
upon business needs and objectives
2
SP 12 Define Test Policy A test policy aligned with the business
(quality) policy is defined based on the
test goals and agreed upon by the
stakeholders
3
SP 13 Distribute Policy to Stakeholders The test policy and test goals are
presented and explained to stakeholders
inside and outside testing
SG2 Establish a Test
Strategy
4
SP 21 Perform a generic product risk
assessment
A generic product risk assessment is
performed to identify the typical critical
areas for testing
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Certification Trends
13
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
IT Service Provider Results
14
0
10
20
30
40
50
60
70
80
90
100
21 Test Policy and Strategy
22 Test Planning
23 Test Monitoring andControl
24 Test Design andSpecification
25 Test Environments
31 Test Organisation
32 Test Training program
33 Test Lifecyle andIntegration
34 Non Functional Testing
35 Peer Reviews
IT Service Provider Sector Results
2015
2014
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Results across all industries
15
54
62
56
40
45
51
45
49
69
60
32
27
34
45
43
36
43
42
25
33
14
12
10
15
12
13
12
9
6
7
0 10 20 30 40 50 60 70 80 90 100
Test Organisation
Test Training Programme
Test Lifecycle and Integration
Non Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
Leve
l 3Le
vel 2
TMMi Level 2 amp 3 Process Areas
Fully Achieved Partially Achieved Not Achieved
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Recognise and embrace the failures
bull The fear of failure and a sense of failure are both emotional responses that need to be managed to bring about success
bull You should think about failure in a different way and create an awareness that failure should be regarded as a positive every time
bull To help you appreciate that not all failures are equal and then explore ways of dealing with and preventing them in future
bull See how failure sits alongside process improvement and how risk fits in all of this
16
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Is this right
17
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
General views on Failure
bull Comparing Aviation to Healthcare
bull Generally a failure to learn from mistakes has been one of the single greatest obstacles to human progress ndash Matthew Syed
bull Studies show we can be so worried about failure that we create vague goals so nobody can point a finger when they are not achieved
bull When we fail we donrsquot get a process improvement because the information about errors problems etc are misinterpreted or ignored This we call a closed loop
bull An open loop does lead to progress because you act on feedback
o If we fear failure we donrsquot even try
o If we donrsquot try we donrsquot take risks
o If we donrsquot take the risk then we donrsquot fail and learn how to improve
18
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Amy Edmondson - Harvard Business School
ldquoLearning from failures is anything but straightforward The
attitudes and activities required to effectively detect and
analyse failures are in short supply in most companies and
the need for context-specific learning strategies is
underappreciated Organisations need new and better ways
to go beyond lessons that are superficialrdquo
19
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
bull Failure has created a bad feeling from a young agebull from childhood we learn that admitting to failure can lead to blame
low marks ridicule etc ndash so weve had 12 years or so of training to fear failure
bull often as we grow up we donrsquot shift to the rewards of learning from failure
bull sometimes we fear failure even before we start - aim low you get low
bull We can fear other peoplersquos view of us and this can lead to a lack of confidence
bull We imagine others care as deeply about our own success and failure as we do personally
bull We fear shame and our responses to this can manifest itself in many ways
So overall failure can and has created negative connotations in many of us
20
Why do we fear failure
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
How we should really see failure
bull Organisations need to make it a ldquosaferdquo place to admit and report failure
bull with the right mind-set the concern that accepting failures will lead to poor quality is not relevant
bull Google X ndash Astro Teller says
bull ldquoYou must reward people for failing If not they wont take risks and make breakthroughs If you dont reward failure people will hang on to doomed ideas for fear of the consequences That wastes time and can damage an organisations spiritrdquo
bull Harvard Business Review research developed ldquoa Spectrum of Reasons for Failurerdquo
bull Recognise that no-one deliberately wants to fail
bull We learn from failure with the intention that we can then improve and learn
21
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Spectrum of failure
22
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
How we should really see failure
Opportunity to
bull Learn and grow from the experience
bull Consider and have a healthy approach to risk
bull Forget perfection ndash ldquofail wellrdquo take risks and learn from them Helps you to find the gaps and take actions to fix
bull Be resilient and keep trying ndash eg practise at music to get better ice skating ski jumping etc
bull Recognise that young people donrsquot naturally fear failure so they can learn
bull In a UK school they have a ldquofailure weekrdquo to de-mystify the word Failure happens Then have a ldquoblow your trumpet weekrdquo to recognise and celebrate success and achievements
Be expert learners not just experts
True success is not avoiding failure but learning what to do with it
23
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Wimbledon High School
24
FAILURE WEEKA top girls school is planning a failure week to teach pupils to embrace risk build resilience and learn from their mistakes The emphasis will be on the valueof having a go rather than playing it safe and perhaps achieving less Pupils atWimbledon High School will be asked how they feel when they failThe headmistress Heather Hanbury said she wanted to show it is completelyacceptable and completely normal not to succeed at times in lifeldquoMs Hanburys pupils achieve some of the highest exam scores - but from Monday they will be invited to focus on failure There will be workshops assemblies and activitiesfor the girls with parents and tutors joining in with tales of their own failuresThere will be YouTube clips of famous and successful people who have failed along the way and moved onThe emphasis will be discussions on the merits of failure and on the negative side of trying too hard not to failCourage in the classroomMs Hanbury told BBC News that she had placed a great emphasis on developing resilience and robustness among the girls since she arrived at the school fouryears ago The girls need to learn how to fail well - and how to get over it andcope with it she said Fear of failing can be really crippling and stop the girlsdoing things they really want to doThe pupils are hugely successful but can sometimes overreact to failure even though it can sometimes be enormously beneficial to them We want them to be brave - to have courage in the classroom she added Wimbledon High is an independent school part of the Girls Day School TrustGDST chief executive Helen Fraser said Resilience is so important in working life these daysWimbledon High School is showing how making mistakes is not necessarily a bad thing that it is fine to try - and fail - and then pick yourself up and try again -or as Samuel Beckett said fail better
Blow your own trumpet weekrun by Wimbledon High School (motto
Stepping in Striding out) aims to help its over-pressured high-achievingpupils who find it difficult to accept that they have been successful Head Heather Hanbury explains Some teenagers will continue to push themselves never recognising when theyve done enough The school has also run a successful failure week
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Key quotes
25
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
What does success involve
26
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Project Retrospectives Can Help
Definition of Retrospective Relating to or thinking about the past
bull Where were the failures ndash any failures
bull Identify what worked well and what needs improvement
bull Led by an independent facilitator
bull Held in a safe place ndash quiet not disturbed
bull Not a witch hunt ndash looking at processes
bull Plan the event ndash issues questions at the start of the project
bull Have the right people involved
bull Record results in open area allocate and monitor actions
bull Use the results in future projects
bull Be honest
27
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Retrospectives
Norm Kerthrsquos Prime Directive
Regardless of what we discover we must understandand truly believe that everyone did the best job he orshe could given what was known at the time his orher skills abilities the resources available and thesituation at hand
28
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Emotional Histogram
29
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Richard Branson ndash His views
Irsquove been failing for as long as I can remember In fact Irsquove
been failing even longer than that ndash I fell over many times as a
baby before learning how to walk The pattern has continued
into adulthood and my life as an entrepreneur and I have
learned and loved every step of the way Wersquore celebrating
failure all month at Virgin and teamed up for a Hangout on
How to Turn Failure into Success
30
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Thomas Edison - Quote
31
American Businessman and Inventor Among other things he invented the Electric Light Bulb
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Lessons to learn
32
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Everyone should aim to
33
TRANSFORM THE FEAR OF FAILURE
TO THE ACTION OF SUCCESS
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
ldquoI Dare you to Failrdquo
ldquoIt does not mean we should aim at failure rather that creativity isa journey that involves taking wrong turns along the wayrdquo
34
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Summary
Looked at
bull The challenges you face
bull Examples of client benefits
bull How to move forward with TMMi
bull TMMi trends
bull Donrsquot fear failure ndash embrace it
35
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi 모델을 활용한 Quality Process 개선 사례
HH Cho hhchostacokr
Session 5
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
3
Question 1
Are you aware of TMMi using small i
and CMMI using Large I
No Yes
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
4
Question 2
Do you know why TMMi using small i
and CMMI using Large I
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
5
Question 3
Are you new to SIX-SIGMA(DFSS) LEAN or Lean-Six Sigma
No Yes
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
7
품질 이슈
Quality can make the difference between excellence and disaster
Toyota Recall(2009)
BPrsquos Oil Spill(2010)
Accident Reason Root Cause Quality Focused
O-ring Flaw
Lack of O-ring Testin a very low temperature
NASA disregarded warning from engineers
Lack of factual based decision making
Sudden unintended acceleration
Lack of quality controlfor the TPS and so on
Growth-oriented policies
Safety of drilling equipment (Blowout Presenver) will not
operate hydraulic leaks and battery failure
Poor test for cement borehole and so on
Lack of systematic approachLack of factual approach to
decision making
ChallengerExplosion(1986)
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
8
SW 품질과 비즈니스 영향력(1)
lt Forrester의 북미(US Canada) 유럽(the UK France and Germany) IT 종사자 Survey gt
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
다양한
SW 제품
개발
다양한직무
[ Base 336 product development and IT professionals involved with software development ]
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
SW 품질과 비즈니스 영향력(2)
There is tight link between SW Quality and Business Impact
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
ldquoSW 코드 결함으로 가장 큰 영향을 받는 것은 무엇인가rdquo
Customer satisfaction
Product release scheduletime-to-market
Brand image
Company revenues
Product uptime
Donrsquot have an opinion
None of the above
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
10
비즈니스 목표와 SW Quality 노력
Software Quality(Risks) Have a Direct Impact on Business Success
Software Integrity Risk Report April 2011 Forrester Research
7
7
11
13
7
rsquo
rsquo
3
2
3
4
3
15
19
2
5
[ Base 336 product development and IT professionals involved with software development ]
ldquo품질 개선 활동을 추진하는 주요 이유가 무엇인가 rdquo
분명한비즈니스
목표
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
왜 테스팅이 개선되어야 하나
Software industry has invested substantial effort to improve the quality of its products but the software industry is still far from zero defects
[ Source The Standish Group Chaos Manifesto 2013 ]
Challenged
Succeed
Failed
60
50
40
30
20
10
0
2004 2006 2008 2010 2012
We can no longer imitatethe three monkeys Hear no failures See no failures Speak no failures
Testing often takes 30 - 40 of project costs
IT Projects are harder than climbing Everest
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
SW 테스팅을 개발 앞 단계에서부터~~
Early removal of defects contribute to mitigating risks and improving SW quality
[ Source Software Integrity Risk Report April 2011 Forrester Research ]
8
10
14
10
4
22
27
2
ldquo개발 프로젝트의 성공에 가장 큰 영향을 줄 이슈는 무엇인가rdquo
1
1
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
프로세스 개선 프로그램
많은 조직들이 품질과 생산성 향상을 위해
전사관점의 프로세스 개선 프로그램을 수립함
(company-wide process improvement program)
이러한 프로그램은 종종 많은 양의 프로세스 문서를 만들고
결국엔 부담이 되어 무시되어 버리는 결과를 초래함
(become a burden or are ignored completely)
In many cases the organization is left
with very little benefit to show for its efforts
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
린 식스 시그마 Timeline
1900 1930 1950 20001980 1990
Just-in-Time
포드- Assembly line- Flow Concept
Walter ShewhartSPC 개발
도요타 TPS- Flow concept from ford- Pull concept from grocery store
SPC TQM MotorolaIntroducesSix Sigma
Deming SPC 확산14Points 제안
AlliedSignalGE Adapt LSSto BusinessProcesses
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
CMM(I) SPICE TMM(i) Timeline
1991 1993 20091995 1996
CMM V10 CMM V11 ISO 15504(SPICE)
first draft released
TMM
TMMi V20
(Level 2 amp3) released
2015
ISO 33063 released
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
개선을 위한 두 가지 Approach
벤치마크 정보 활용 =gt 모델 기반
bull Compare yourself to others who have similar strategies
(but not necessarily similar products)
bull Top-down strategy of borrowing from others to learn something new
bull Decomposes the improvement needs into processes
문제해결을 위한 Flexible way
bull Improve the quality of your processes by eliminating waste and variation
bull Bottom up strategy where improvement begins at the work-team level
bull Used to solve specific problems
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Model ndash Maturity Level
bull Test Policy and strategybull Test Planningbull Test Monitoring and Controlbull Test Design and Executionbull Test Environment
L2 Managed
bull Test Organizationbull Test Training Programbull Test Lifecycle and Integrationbull Non-functional testingbull Peer Reviews
L3 Defined
bull Defect Preventionbull Test Process Optimizationbull Quality Control
L5 Optimization
Level 1 Initial
bull Test Measurementbull Product Quality Evaluationbull Advanced Peer Reviews
L4 Measured
High Maturity
Infrastructure for Process
Improvement
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Maturity Level(45)
Level 4 Measured
Definition
Features
Visibility
Testing is a thoroughly defined well-founded and measurable process
bull Testing is perceived as evaluationbull Measuring Test Performance and costbull Measuring Product Qualitybull Measuring Review Result
Quality and process efficiency are being statisticallymeasured through the whole development lifecycleA test measurement program supports predictionsrelating to test performance and cost
Support fact-based decision
making
IN OUT
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi Maturity Level(55)
Level 5 Optimization
Definition
Features
Visibility
Optimized test process is focused on defect prevention
bull Continuously improvedbull Fine-tuning methods and techniquebull Tool support as much as possiblebull Process reuse
Test Process is optimized and continually improvingits processes
Six Sigma(DFSS) amp Leanis helpful for
CMMI and TMMi Level 5
IN OUT
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
The Benefits of TMMi
An IT organization that reached TMMi Level 3 as one of the first testing organizations worldwide reported a higher Defect Detection Percentage during the system test
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
(DDP)
75
70
65
60
55
50
1st year 2nd tear 3rd year 4th year
DDP(Defect Detection Percentage)
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
The Benefits of TMMi
An IT organization at TMMi Level 2
reported better predictability of the
testing process
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
A Financial organization reached
TMMi Level 2 reported the
improvement of the DDP
()(DDP)
50
40
30
20
10
0
1st year 2nd tear 3rd year
Defect Detection Percentage of the system testDeviation of test time spent versus estimated test time (percentage)
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Ⅰ The importance of (SW) Quality
Ⅱ Introduction to TMMi
Ⅲ SW Quality Improvement using TMMi
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Case Study of Defense Industry
Company L Company H
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이 높음
bull Continuous improvement activities
bull Perform a step-by-step risk analysis(from generic risk analysis)
Strong Points
bull CMMI L5 bull SIX SIGMA(DFSS) LEAN SIX SIGMA
bull 경영진의 품질 및 테스트 인식 수준이높음
bull Continuous improvement activities
bull Well known about process (Recently enhanced dev quality control gate)
bull Some members do not well known about test processes
bull Lack of risk tracking bull Lack of test log managing
Weak Points
bull Lack of dedicated test organizationbull Lack of professional test expertisebull Lack of risk tracking
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Case Study of Defense Industry
Similarities and differences of both companies
Company L Company H
CMMI Level 5 Development CMMI Level 5
TMMi Level 3 Testing Level TMMi Level 3
Common risk analysis based on quality attributes
Risk Based Testing Requirements based risk analysis
Define new procedures to derive Test conditions
Test conditionApply to the existing test Design
documents
Newly create new Master Test Plan(MTP)
Test PlanUpdated Original
Master Test Plan(STP)
V-Model-based testing activitiesIntegration with Dev
lifecycleV-Model-based testing activities
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
CMMI Process Areas that Support a TMMi Implementation
CMMi Level Process Area
Level 5 (Optimizing)
Org Innovation and Deployment
Causal analysis and Resolution
Level 4(Quan Managed)
Organizational Process Performance
Quantitative Project Management
Level 3 (Defined)
Requirements Development
Technical Solution
Product Integration
Verification
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Integrated Supplier Management
Risk Management
Decision Analysis and Resolution
Level 2 (Managed)
Requirements Management
Project Planning
Project Control and Monitor
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Process Area TMMi Level
Defect Prevention
Level 5 (Optimization)
Test Process Optimization
Quality Control
Test Measurement
Level 4 (Measured)
Product Quality Evaluation
Advanced Peer Reviews
Test Organization
Level 3 (Defined)
Test Training Program
Test Lifecycle and Integration
Non-functional testing
Peer Reviews
Test Policy and strategy
Level 2 (Managed)
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
L 사 사례 ndash WHY
L 사가 TMMi 기반 테스트 프로세스 개선을 추진한 이유
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf they donrsquot do something they wonrsquot survive
In order to maintain a leader amp
have a good position in bidding situation
(국방산업 분야 최초 인증)
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
L 사 사례 ndash 프로세스 개선 활동
TMMi based Test Process Improvement March 2015 LIG Nex1
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
MRA update
MRA Manufacturing Readiness Assessment CMMI Capability Maturity Model Integration TMMi Test Maturity Model integration
RampD 품질경영시스템
국방 정책
SW Process Update
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
L 사 사례 - SW 테스트 프로세스 구현
TMMi based Test Process Improvement March 2015 LIG Nex1
Establish Master Test Plan
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Establish Test Performance Indicators
Perform integrated test planning activities
rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
Try to separate test organization
rarr SW Test and Whole System Test
Develop test Job description amp Career Path
Develop test training Program
Develop Test organizationand training program
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
L 사 사례 - Testing Activities
TMMi based Test Process Improvement March 2015 LIG Nex1
bull Static Testing tool
- QAC++ CodeSonar
bull Static Testing tool
- QAC
bull Adopt SW Incident management tool
- QualityCenter
bull introducing dynamic testing tools
- VectorCAST LDRA
bull SW Resource Management tool
- SharePoint
bull SW Dynamic Test tool
- DT10
bull SW Static test tool
- CodeSonar(USB)
bull SW Test management tool
- Jama
bull SW Build Automation tool
- Jenkins
bull SW Static test tool
- QAC++(USB) QAV
SW Coding guideline
Static Testing Quality
Management
SW ReliabilityTest
TMMi based Process
Implementation
TMMi based Process Institutionalization
bull SW Coding guideline
bull SW Testing Pilot project
bull Extend SW Testing Scope
bull SW Reliability Test(Organizational)
bull Build SW testing process
bull TMMi Level 3
bull TMMiinstitutionalization
bull Independent testing
bull Setup Partnerrsquos SW Process
bull Educate Partnerrsquos SW reliability test
bull Perform Partnerrsquos SW reliability test
bull Develop SW Quality Training Program and Train(3 classes)
bull Develop SW Quality Training Program and Train( 6 classes)
Quality Management Team
Development Team Leads the Activities and Quality Team is supportive
Suppliers are also applied
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi 적용 효과 - L 사
SW 개발 수명주기 앞부분에서 결함 검출율 향상
TMMi based Test Process Improvement March 2015 LIG Nex1
- The defect density of the coding and unit testing and SW integration testing phase increased significantly
- The defect density of System integration testing is reduced
Applied TMMi based Processes7 Projects
Not applied TMMi Processes33 Projects
of Defects KLOC
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi 적용 효과 - L 사
Organizational Pride
ldquoThe successes weve had in process improvement have been a source of great pride for our quality assurance teamrdquo
사내에 품질 경영을 위한 기본적인 품질관리 프로세스가CMMi 등으로 정착이 되어 있는 상태에서
테스트를 위한 프로세스를 보완해품질관리의 성과가 더욱 빛나도록 함
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
H 사 사례 ndash WHY
Why did Company L apply Process Improvement using TMMi
Everyone else is doing it An executive heard about it or read about it somewhereA customer demands itIf we donrsquot do something we wonrsquot survive
A compelling reason for changeAt the VOC meeting
The CEO announced to initiate TMMi
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Applied
H 사 사례 - Process Improvement
RampDProcess HW based
ProcessSW based Process
HWSW Integration Process
Measurement and Continuous Improvement
Measurement and Continuous Improvement
Defense embedded SW guideline
SW related contents update
TMMi based Process update
DFSS Design for Six Sigma
Defense Industry Policy(Quality Management System)
SW Process Update
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
H 사 사례 - SW 테스트 프로세스 구현
Establish Master Test PlanDevelop Test organization
and training program
Establish Test Policy and Strategy Strengthen Test Design Activities
Organizational Test Policy
rarr Product Risk based test strategy
Reused Test Performance Indicators
Update original Master test plan(STP)rarr Integrate Level test plan
Establish product risk analysis activities
and testing strategy
Clarification of the testing exit criteria
Adopt formal test techniques
Make test design specification templates
Identify early test item and design
for the dynamic test
SE WPL and SW WPL have Responsible
for Testing and Development work
Develop test Job description amp Career Path
Develop test training Program
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi 를 효과적으로 적용하기
To effectively apply TMMi implies reflecting on the realities of your business environment and applying the model in a thoughtful appropriate manner
bull Identifying problems as objectively as possible and tieprocess improvement to your business goals
bull Tailoring (interpreting) TMMi to fit your context and needs
bull Not forcing inappropriate solutions simply because a practice is identified
in the TMMi model
Use TMMi as a Guide not as a Rule Book
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi 기반 프로세스 개선의 핵심요소
A compelling reason for change
Leadership of the change effort by the top executive
Informed commitment of the top management team
Designation of a primary change agent and an adequate mandate for change
Sound performance measures that drive change
Without these itrsquos very likely that you will experience
a ldquofalse startrdquo of your process improvement program
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
테스트 프로세스 개선의 Manifesto
Detailed Process
Process oriented
Templates
Quality assurance
Model driven
IP3 Deployment oriented
IP2 Best Practices
IP4 Peer reviews
IP1 Flexibiltiy
IP5 Business driven
[ Source Erik van Veenendaal and Jan Jaap Cannegieter The Little TMMi UTN Publishers 2011 ]
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
Conclusions
1 TMMi can be a good tool for test process improvement
2 TMMi has similar structure with CMMI so a company using CMMi can
easily apply TMMi practices
3 Needs a compelling reason for change
Just trying to get to TMMi Leve3 without understanding the
business context will be fail in the short or long term
4 The improvement team must understand the business problem
5 As TMMi is a relatively recent model the number of publications
about benefits in practice is limited However applying the TMMi maturity
will improve test process and have a positive impact on product quality
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
TMMi based improvement case study of
software reliability testing process at LIG Nex1
LIG Nex1 김명호 myonghokim91gmailcom
LIG Nex1 신영섭 busigeenavercom
Session 6
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 회사 소개
2 LIG Nex1 테스트 프로세스 개선 현황
3 소프트웨어 신뢰성 시험 개요
4 TMMi 기반의 테스트 Process 구축
5 소프트웨어 신뢰성 시험 프로세스 개선
6 결론 및 향후 계획
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 회사 소개
3
LIG넥스원은 정밀유도무기 등 방위산업
전 분야에 걸친 최첨단 무기체계를
연구개발생산 기업임
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
1 회사 소개
4
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
5
2 테스트 프로세스 개선 현황
국방 정책 및 Global 모델(CMMI TMMi 등)에 따라 지속적인 프로세스 개선활동을 수행하고 있음
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
6
2 테스트 프로세스 개선 현황
rsquo07년 부터 SW 신뢰성을 향상을 위한 프로세스 인프라 교육 등의 활동을 수행함
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
3 소프트웨어 신뢰성 시험 개요
7
무기체게 소프트웨어는 ldquo무기체계 소프트웨어 개발 및 관리 매뉴얼rdquo을 기준으로시험을 수행함
bull ldquo소프트웨어 신뢰성 확보 활동rdquo이란 소프트웨어의 잠재적 결함을 최소화시키기 위한
제반 활동(신뢰성 확보계획 수립 검증 및 확인 신뢰성 시험 등) 을 말한다
bull ldquo소프트웨어 신뢰성 시험rdquo 이라 함은 소프트웨어 코드가 일으킬 수 있는 결함을 사전에
식별하여 제거하기 위한 시험을 말하며 정적 시험 및 동적 시험으로 구분한다
1) ldquo소프트웨어 정적(Static) 시험rdquo이란 소프트웨어를 실행하지 않은 상태에서
잠재적인 결함을 검출하는 시험을 말하며 코딩 규칙(Coding Rule) 검증 취약점
점검 및 소스코드 메트릭 점검으로 구성된다
2) ldquo소프트웨어 동적(Dynamic) 시험rdquo이란 소프트웨어의 요구사항 및 구조를
기반으로 소프트웨어를 실행하면서 기능의 정상 동작 여부를 확인하는 시험을
말한다
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
3 소프트웨어 신뢰성 시험 개요
8
소프트웨어 신뢰성 시험의 시험 기준은 지속적으로 높아지고 있음
구분 rsquo11년 rsquo14 년 rsquo16 년
코딩규칙
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 방위사업청 코딩규칙 적용- 공통 규칙 45개- C 언어 전용 5개- C++ 언어 전용 10개
bull 국제 표준을 사업별 조정 및 합의- C 언어 MISRA C (143개 규칙)- C ++ 언어 MISRA C++ (228개 규칙)- C 언어 Microsoft (62개 규칙)- JAVA Oracle
bull 방위사업청 코딩규칙은 유지
취약점bull 실행시간 오류검출 목록
CWE-658659 bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659
bull 사업별로 선별 적용
bull 실행시간 오류검출 목록 CWE-658659660bull CWE 목록 중 도구가 지원하지 않는 항목에 대
해서는 대안과 사유를 제출하고 협의하여 결정
소스코드
메트릭bull 주석률 30 이상 bull 주석률 30 이상
bull 주석률은 코딩규칙으로 이동
bull Cyclomatic Complexity 20 이하bull Number of Call Levels 6 이하bull Number of Function Parameters 8 이하bull Number of Calling Functions 8 이하bull Number of Called Functions 10 이하bull Number of Executable Code Lines 200 이
하
bull 정적 시험 분야
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
3 소프트웨어 신뢰성 시험 개요
9
rsquo11년 rsquo14 년 rsquo16 년
bull 임무중요도 기능안정성 및 통제능력 사용빈도에 따라 목표값 설정 후구조기반 동적 시험수준 결정
bull NA Statement Branch MCDC
bull 국제기준 및 규격(MIL-STD-882E DO-178B IEC 61508 ISE 26262 등)을 참고하여 시험대상 결정 후위험수준별 구조기반동적 시험 수준 결정
bull NA Statement Branch MCDC
bull 결함 발생빈도 영향성 및 제어가능성에 따라 수준별요구사항기반 동적 시험 수행
bull Statement Branch MCDC ( NA 없음)bull 단 국제표준(MIL-STD-882E DO-178 IEC 61508 ISE
26262 등) 을 적용하여 개발한 경우는 해당 표준을 따를 수 있다
bull 동적 시험 분야
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
3 소프트웨어 신뢰성 시험
10
bull Open Source Software
rsquo11년 rsquo14 년 rsquo16 년
- -
bull 소프트웨어 신뢰성 시험 대상
bull 상세 설계 단계에서 사용필요성 및 신뢰성 확보 제시하여 승인 시 제외 가능
bull 오픈 소스 사용 시는 라이선스 조항을 반드시 확인하여야 하며 소스코드 공개 의무가 있는 라이선스는 사용이 불가함
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
4 TMMi 기반의 테스트 프로세스 구축
11
Optimization1 결함 예방2 테스트 프로세스 최적화3 품질제어
Measured1 테스트 측정2 제품 품질 평가3 발전된 동료 검토
Defined1 테스트 조직2 테스트 훈련 프로그램3 테스트 수명주기amp통합4 비기능 테스팅5 동료 검토
Managed1 테스트 정책amp전략2 테스트 계획3 테스트 모니터링amp제어4 테스트 설계amp실행5 테스트 환경
Level 5
Level 4
Level 3
Level 2
Initial
Level 1
TMMi 공식 인증 시작(2010년~)
bull 조직 차원 4개 영역 테스트 정책 및 전략 시험조직 테스트 훈련 프로그램 테스트 수명주기 amp 통합
bull 프로젝트 차원 6개 영역 테스트 계획 테스트 모니터링amp제어 테스트 설계amp실행 테스트 환경 비기능 테스팅 동료
검토
LIG넥스원은 Level 2 Process Area 5개 Level 3 Process Area 5개 총 10개영역을 적용함
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
4 TMMi 기반의 테스트 프로세스 구축
12
LIG넥스원에 구축된 SW 테스트 프로세스를 TMMi 영역 별로 도식화하면다음과 같은 연관 관계가 있음
개발과 시험 수명주기 통합 모델
시험 정책 및 전략
시험 조직
시험 훈련 프로그램
시험 계획
시험 환경
시험 설계 및 수행
비기능 시험
시험 모니터링 동료 검토
PA21 Test Policy and Strategy
PA33 Test Lifecycle and Integration
PA23 Test Monitoring and Control PA35 Peer Reviews
PA31 Test Organization PA22 Test Planning
PA32 Test Training Program
PA25 Test Environment
PA24 Test Design amp Execution
PA34 Non-Functional Testing
조직차원 정의
프로젝트 차원 정의
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
4 TMMi 기반의 테스트 프로세스 구축
13
TMMi Level 2~3의 10개 Process Area는 사내 표준 프로세스에 포함되어 있음
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
14
소프트웨어 시험 프로세스는 방위사업청 규정 및 MIL-STD 기반으로 CMMI 레벨5및 TMMi 레벨 3 요건을 충족시키는 총 33종의 표준 프로세스가 정의되어 있음
4 TMMi 기반의 테스트 프로세스 구축
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
15
기본 방침은 V 모델 기반으로 개발 초기부터 신뢰성 시험활동을 수행토록 함
4 TMMi 기반의 테스트 프로세스 구축
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
16
5 소프트웨어 신뢰성 시험 프로세스 개선
배경 소프트웨어 신뢰성 시험 기준 변경
추진 방법
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
Step 5 3단계와 4단계를 통해 개선 및 추가되어야 할 활동을 통합하여To-Be 프로세스를 구축
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
17
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 1 신뢰성 시험의 주요 변경사항 분석 및 이슈 도출
총 16개 주요 변경 사항 및 이슈
- 공통 분야 8개
- 정적 시험 분야 4개
- 동적 시험 분야 4개
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
18
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 2 도출된 이슈를 Process People Product 및 Environment 관점에서 개선 과제 도출 및 우선순위를 고려한 개선 계획 수립
개별 사업별 해결 3개
과제 우선 순위
- A 즉시 실행
- B rsquo17년 실행 과제
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
19
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 3 개선 과제 중 신뢰성 시험에 대한 As-Is 프로세스를 정의 및문제점 도출
Process Modeling BPMN 20
개발 단계별 Process 정의
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
20
5 소프트웨어 신뢰성 시험 프로세스 개선
Step 4 개정된 규정에 의해 추가되어야 할 활동을 정리
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
21
6 결론 및 향후 계획
To-Be Process 특징
첫째 신뢰성 시험을 시험 항목별로 연관성을 분석하여 시험 프로세스를
표준화하고 시험 항목별 목표치를 설정하여 재작업을 최소화
둘째 개발 초기부터 전문 부서의 검토 단계를 추가하여 시험 단계에 발생
할 수 있는 문제를 사전에 최소화할 수 있도록 하였다
셋째 신뢰성 시험 전문 조직의 역할을 늘리고 개발 조직과의 협업
프로세스를 구축하여 시험 생산성을 향상하였다
향후 계획
프로세스 개선 효과 측정 테스트 생산성 결함 검출률 등
L4 amp L5 기반 프로세스 개선 동료 검토 생산성 향상 정적 시험 목표치 설정 등
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr
대한민국 SW 품질을 책임집니다
대한민국 SW 가치 증진을 선도하고 산업을 리딩하는SW테스팅 전문기업 네트워크
회원사 명단
회장사
회원 가입 문의
와이즈와이어즈
임원사 MDS테크놀로지 STA테스팅컨설팅 어니컴 와이즈스톤 지티지 파수닷컴
회원사 가온테스팅 두루이디에스 링크제니시스 솔루션링크 슈어소프트테크
씨엔큐소프트 아이오에이솔루션 아이지오 아주큐엠에스
에스이앤티소프트 에이취큐텍 엠스텍 지텍에스 큐머드
한국전력공사 전력연구원 NIPA SW공학센터
컴퓨터월드ITDAILY
한국SW테스팅협의회에서는 관련 전문 기업 및
조직의 참여를 기다리고 있습니다
(자문) 숭실대학교 SW학부 (박진호 교수)
한국SW산업협회 배범진 책임
연락처 bbj99sworkr