Portugal

Problems of CMMI® Implementations

Recommendations to Avoid them

and Achieve High Maturity

Isabel Lopes Margarido

PhD Student Researcher

Faculdade de Engenharia da Universidade do Porto

2013-10-18, Lisboa

João Pascoal Faria

FEUP/INESC

Marco Vieira

FCTUC

Raul Moreira Vidal

FEUP

2 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

introduction

methodology

problems and recommendations

conclusions

agenda

3 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

introduction method problems conclusions

4 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

introduction method problems conclusions

5 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

objectives

gain additional knowledge about CMMI

introduction method problems conclusions

6 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

objectives

understand problems and difficulties

gain additional knowledge about CMMI

introduction method problems conclusions

7 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

objectives learn solutions to avoid them

understand problems and difficulties

gain additional knowledge about CMMI

introduction method problems conclusions

8 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

motivation

typically organisations using CMMI improve performance

many programs failed in CMMI level 5 organisations

SEI was concerned with

high maturity

introduction method problems conclusions

9 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

high maturity levels dependencies

dependent on lower maturity levels

maturity level (ML) 2 M&A

introduction method problems conclusions

10 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

high maturity levels dependencies

dependent on lower maturity levels

maturity level (ML) 2 M&A

building blocks for ML4: QPM, OPP (PPM and PPB)

introduction method problems conclusions

11 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

high maturity levels dependencies

dependent on lower maturity levels

maturity level (ML) 2 M&A

building blocks for ML4: QPM, OPP (PPM and PPB)

knowledge base for quantitative continuous improvement at

ML5 (CAR, OPM)

introduction method problems conclusions

12 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

case study (published in QUATIC, Lopes Margarido et al., 2012)

3 organisations appraised at CMMI

Dev ML5

– 2 organisations, 1 business unit

real problems and difficulties from industry

– documents, tools, interviews

– how CMMI implementation was conducted and the

processes were defined

– what processes, metrics and tools were developed

– how people were actually using them

introduction method problems conclusions

13 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

analysis

analysed an SEI report (McCurley and

Goldenson, 2010)

verified which problems were common

to the case study, literature and SEI survey

verified which of the recommendations were sustained by

literature and SEI survey

introduction method problems conclusions

14 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

problems (P) and recommendations (R)

entry conditions

P1: avoid underestimation

R1: plan: maturing levels, analysing and understanding HML,

building and maturing PPB and PPM

P2: don’t begin the house by the roof

R2: ML 2 and 3 need to mature

P3: understand quantitative nature of level 4

R3: involve statistician

R4: six sigma

R5: review goals

introduction method problems conclusions

15 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

process definition and implementation

P4: copied processes

R6: reflect organisation culture

R7: involve experts and process users

P5: multicultural environments

R8: share processes, lessons learnt

P6: impose processes

R8 and R9: goals specific for business units related to

organisation business goals

R10: indicators at different report levels

introduction method problems conclusions

16 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

process definition and implementation

P7: dissemination problems

R11: commitment from entire organisation

R12: training contents, specialised training

R13: coaching and monitoring

P8: lack of institutionalisation

R14: top management set goals for gradual

institutionalisation, monitor and reward

R13 and R15: give time for metrics and processes to mature

introduction method problems conclusions

17 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

metrics definition

P9: meaningless uncorrelated metrics

P10: metrics definition (collect and analyse data)

R16: use goal question metric or equivalent

R17: unambiguous, repeatable, understandable

R18: size metrics according with work product

R19: interpret in context

R20: variables normalisation

introduction method problems conclusions

18 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

metrics definition

P11: first data uncorrelated

P12: metrics categorisation

P13: baselines not applicable

R21: several cycles

R22: let PPM and PPB become stable

R23: categorise data

R24: aggregate normalised data

introduction method problems conclusions

19 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

metrics usage

P14: abusive elimination of outliers

R25: quarantine

R26: maintain data points that are unique but recurrent

P15: not all projects are measurable

R27: specific based measures normalised to get derived

measures; R14

introduction method problems conclusions

20 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

metrics usage

P16: effort estimates

R28: use expert judgment when needed

R29: any related historical data

R30: iterative planning, real time sampling

P17: people behaviour

R12, R13 and R31: personal data not used to evaluate

people

introduction method problems conclusions

21 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

tools setup

P18: tools setup

R32: give time for tool setup

R33: do not use data collected when defects affect metrics

P19: overhead

P20: tools requirements

R34: only collect necessary data

R35: automated and unperceptive data collection

R36: discipline people, change mentality

introduction method problems conclusions

22 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

problems analysis

59% found in the case studies, literature and survey

several problems shared between the studied organisations

37,5% found in literature were found in the case studies

53,3% found in the survey were found in the case studies

16,7% of those also found in literature

introduction method problems conclusions

23 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

problems analysis

59% found in the case studies, literature and survey

several problems shared between the studied organisations

37,5% found in literature were found in the case studies

53,3% found in the survey were found in the case studies

16,7% of those also found in literature

“the percentage of problems shared in more than one

organisation/source indicates that they can occur when

implementing HML, so organisations should be aware of

them.” (Lopes Margarido et al., 2013, submitted for

publication to SQP)

introduction method problems conclusions

24 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

summary and future research

wide variety of implementation methods

variance of performance results

SCAMPI:

– evaluates a sample

– does not evaluate performance

future research

– framework to evaluate quality of implementation of

CMMI practices

– definition published in PROFES 2012 (Lopes Margarido

et al., 2012)

introduction method problems conclusions

25 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

think about...

introduction method problems conclusions

26 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

questions

http://paginas.fe.up.pt/~pro09003

partially sponsored by

27 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

references

• M. Campo. “Why Maturity Level 5?”, Crosstalk, January/February

2012. 15-18.

• C. Hollenbach and D. Smith. (2002) A portrait of a CMMI level 4

effort Systems Engineering. 52-61.

• P. Leeson, "Why the CMMI® does not work," presented at the

SEPG Europe, Prague, Czech Republic, 2009.

• R. Radice, "Statistical Process Control in Level 4 and Level 5

Software Organizations Worldwide," presented at the Software

Technology Conference, 2000.

• M. Schaeffer, "DoD Systems Engineering and CMMI," presented

at the CMMI Technology Conference and User Group, 2004.

28 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

acronyms

CMMI – Capability Maturity Model Integration

M&A – Measurement and Analysis

ML – Maturity Level

OPM – Organisational Performance Management

OPP – Organisational Process Performance

P – Problem

PPB – Process Performance Baselines

PPM – Process Performance Models

QPM – Quantitative Project Management

R - Recommendation

SEI – Software Engineering Institute

SQP – Software Quality Professional

29 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All

Rig

hts

reserv

ed.

Isabel Lopes M

arg

arido,

18

th o

f O

cto

ber

2013,

Lis

bon

images

http://api.ning.com/files/hpf*xOTebDs- F23o6FETZ3j*3sNiONWjfXjTJCzprPjU5bS1

WJoGgWBjMPIOiQkm3SbZ41ijncrJ4K2aT-6dM9QURwHK3led/Dissemination2.jpg -26-06-2010

http://blog.pmtech.com.br/wp-content/uploads/Square-Paradox.jpg – 29-04-2011

http://www.signsexpressshop.co.uk/prodpics/1103.gif – 29-04-2011

Benjamin Haas/Shutterstock, http://cynthiayildirim.posterous.com/how-can-we-measure-the-size-of-the-

universe – 29-04-2011

http://ryanstephensmarketing.com/blog/wp-content/uploads/2009/10/one_size_fits_all.JPG

http://evolvingwe.com/wp-content/uploads/2010/11/image3.png – 29-04-2011

http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010

http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_small.jpg – 02-05-2010

http://www.articulate.com/rapid-elearning/wp-content/uploads/2008/08/summary-objectives450.gif –

adapted, 25-05-2011

http://www.braxtechconsulting.com/Portals/22771/images/cmmi_certificate.jpg.png - 10-04-2012

http://erikhatch.org/wp-content/uploads/2012/02/no_time_1_.jpg - 31-08-2012

http://blog.inktechnologies.com/wp-content/uploads/2012/06/Statistics.jpg - 31-08-2012

http://images.yourdictionary.com/images/definitions/lg/erase.jpg -31-08-2012

https://earlychildcare.files.wordpress.com/2009/09/child-misbehaving.jpg - 31-08-2012

http://www.thaiworldview.com/jpg/img024.jpg - 31-08-2012

http://image.shutterstock.com/display_pic_with_logo/5880/5880,1268065242,15/stock-photo-

background-concept-wordcloud-illustration-of-scientific-method-research-glowing-light-48226942.jpg -

02-09-2012