Embed Size (px)
DESCRIPTION
Website SIGIST: TMMi, the successor of TMM, was first published in 2008. Since then, a lot of organisations have used it to assess or improve their test processes. Jan Jaap Cannegieter used TMMi in Telecom, public services, finance and utility. Based on over 25 assessments conclusions about the level of testing in The Netherlands can be drawn: - Which areas of testing are well developed? - Which areas of testing are not well developed? - What conclusions can be drawn based on these data? These questions will be answered in the presentation. The main conclusion will be that those areas that can easily be influenced by testers are quite well developed; test process improvement is something done by testers in their cave! The areas where other stakeholders need to participate are not that well developed. So to improve testing we need to involve other stakeholders better. The way we should act to involve the relevant stakeholders will be explained and analyzed. Jan Jaap Cannegieter is a leading test, QA, CMMI and requirements expert in the Netherlands. He has 20 years of experience in ICT, starting with software testing and quality assurance in ICT-projects. In the last few years he has completed several TMMi test process improvement, CMMI, SPI and requirements projects. In addition to his assignments, Jan Jaap is Vice President of SYSQA B.V., a company of 180 employees specializing in requirements, software testing, quality assurance and software process improvement. Jan Jaap is the writer of several articles and books, including The little TMMi. Jan Jaap was a member of the TMMi development team and is TMMi accredited lead assessor.
Citation preview
Tester, get out of your cave!
Jan Jaap Cannegieter – Vice President SYSQA B.V. [email protected]@jjcannegieter http
://www.sy
sqa.com/ and
http://w
ww.sysqa.com/expertis
e-it-
quality-management/presentations/
© S
YS
QA
Alm
ere
3
Why are we here?
© S
YS
QA
Alm
ere
4
Agenda
• TMMi in under 10 minutes• Data from 20 TMMi workshops• Conclusions• How to get out of your cave
© S
YS
QA
Alm
ere
6
TMMi – essentials
• Test maturity framework• Benchmark model for assessments• 5 levels• Based on:
• TMM• Evolutionary Testmodel (Gelperin / Hetzel)• CMMI for Development
• In line with ISTQB terminology• Developed by the TMMi-foundation• Public domain• www.tmmifoundation.org
© S
YS
QA
Alm
ere
7
TMMi levels
1. Initial
2. Managed
Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment
3. Defined
Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews
4. Management and Measurement
Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews
5. OptimizationDefect PreventionTest Process OptimizationQuality Control
© S
YS
QA
Alm
ere
8
How I got my data
• Goals• Learn• Presales• Assignments
• 20 organizations• Telecommunication: 2• Government: 7• Financial services: 9• Industry: 2
• Analysis
Results
Fasten your seatbelts
© S
YS
QA
Alm
ere
11
Test policy and strategy - goals
• Establish a test policy• Establish a test strategy• Establish test performance indicators
© S
YS
QA
Alm
ere
12
Test policy and strategy: A: 30% V: 14%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test policy and strategy
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
13
Test planning - goals• Perform a product risk assessment• Establish a test approach• Establish test estimates• Develop a test plan• Obtain commitment to the test plan
© S
YS
QA
Alm
ere
14
Test planning: A: 41%, V: 13%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test planning
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
15
Test monitoring and control - goals
• Monitor test progress against plan• Monitor product quality against plan and
expectations• Manage corrective action to closure
© S
YS
QA
Alm
ere
16
Test monitoring and control: A: 47%, V: 15%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test monitoring and control
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
17
Test design and execution - goals
• Perform test analysis and design using test design techniques
• Perform test implementation• Perform test execution• Manage test incidents to closure
© S
YS
QA
Alm
ere
18
Test design and execution: A: 60%, V: 8%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test design and execution
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
19
Test environment - goals
• Develop test environment requirements• Perform test environment implementation• Manage and control test environments
© S
YS
QA
Alm
ere
20
Test environment: A: 59%, V: 9%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190
10
20
30
40
50
60
70
80
90
100
Test environment
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
21
Overview
Test
pol
icy
and
stra
tegy
Test
pla
nnin
g
Test
mon
itorin
g en
cont
rol
Test
des
ign
and
exec
utio
n
Test
env
ironm
ent
0102030405060708090
100
Per process area
© S
YS
QA
Alm
ere
22
My conclusion
60%: Test design and execution
30%: Test policy and strategy
41%: Test planning
47%: Test monitoring and control
59%: Test environment
Inside
Outside
© S
YS
QA
Alm
ere
23
We are in a cave!
© S
YS
QA
Alm
ere
24
Get out of your cave!
© S
YS
QA
Alm
ere
25
In case you disagree…
© S
YS
QA
Alm
ere
26
Best practices ‘outside of your cave’
• Perform a product risk assessment and establish a test approach with relevant stakeholders
• Obtain commitment for your plan from relevant stakeholders• Plan stakeholder involvement in your project• Monitor stakeholder involvement in your project• Make release advice with people outside of testing• Establish and maintain a test strategy with relevant
management• Define, measure and report test performance indicators to
relevant stakeholders• Establish and maintain a test policy with higher management
(business and IT)Stakeholder m
anagement
Stakeholder management
27
The basics
© S
YS
QA
Alm
ere
28
Stakeholder
A person or group of persons who will be involved in or effected by your test project or improvement project
© S
YS
QA
Alm
ere
29
Stakeholder management
• Stakeholder identification
• Stakeholder analysis
• Stakeholder involvement
© S
YS
QA
Alm
ere
30
Stakeholder aspects
• Organizational level• Power• Business interest• Mindset• Ambition• Behavioral style• Stakeholder type
© S
YS
QA
Alm
ere
31
Type of stakeholders
Demanding Discretionary
Sleeping
Dependent
Dangerous Dominant
Definitive
Urgency Legitimacy
Power
Mitchell, Agle, Wood, 1997
© S
YS
QA
Alm
ere
32
Stakeholdercard – an example
Please think this over
I’ll turn on some inspiring music
© S
YS
QA
Alm
ere
34
© S
YS
QA
Alm
ere
35
Yes, it’s all political
© S
YS
QA
Alm
ere
36
Stuart Reid’s test competence model for testers
Test skills
IT skills
Domain knowledg
e
Soft skills
© S
YS
QA
Alm
ere
37
My test competence model for test consultants & test managers
Test skills
IT skills
Domain knowledg
eSoft skills
Political skills
© S
YS
QA
Alm
ere
38
More interesting graphs
© S
YS
QA
Alm
ere
40
TMMi level by percentage
TMMi Level
84%
10%6%
L1L2L3L4L5
© S
YS
QA
Alm
ere
41
CMMI vs. Non-CMMI
TPG TP TMC TT TO0
10
20
30
40
50
60
70
80
CMMIno CMMI
© S
YS
QA
Alm
ere
42
TMMi score by Industry
TPS TP TMC TDE TE0
10
20
30
40
50
60
70
GovernmentFinanceIndustry
Note: Telecom weg gelaten slechts 2 data punten
© S
YS
QA
Alm
ere
43
Formeel vs. Informeel
14%
86%
FormeelInformeel
© S
YS
QA
Alm
ere
46
