CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved CSE7315M37

CSE 7315 - SW Project Management / Module 37 - Process Appraisal and AssessmentCopyright © 1995-2004, Dennis J. Frailey, All Rights Reserved

Slide 1

CSE7315M37

January 10, 2004

SMU CSE 7315 / NTU SE 584-NPlanning and Managing a

Software Project

Module 37Process Appraisal and

Assessment

Slide # 2 January 10, 2004

CSE 7315 - SW Project Management / Module 37 - Process Appraisal and

AssessmentCopyright © 1995-2004, Dennis J. Frailey,

All Rights Reserved

CSE7315M37

Objective of This Module

• To examine methods of appraising and assessing process maturity




All Rights Reserved

CSE7315M37

Process Appraisal“If you don’t know where you are, a map

won’t help” (1)

An appraisal is a way to determine where you are

(1) Humphrey (textbook)




All Rights Reserved

CSE7315M37

Why Do Appraisals?

Most organizations suffer because they cannot see their problems, not because they cannot solve them.

Most organizations are focused on defining solutions, not on defining problems.

Most organizations tend to solve symptoms, rather than the fundamental, underlying problems.

Why not just solve

the problems?




All Rights Reserved

CSE7315M37

Responding to Symptoms

Problem

Symptom

Symptom

Symptom

Response LongerTerm Cure

Response

TemporaryCure




All Rights Reserved

CSE7315M37

Types of “Appraisals”• Reviews

• Audits

• Capability Evaluations

• Self Assessments

• Other Appraisals




All Rights Reserved

CSE7315M37

Possible Exam Question

Compare and contrast the different forms of appraisal with respect to:–Cost–Length–Who Performs–Objectives–Advantages–Drawbacks–etc.




All Rights Reserved

CSE7315M37

ReviewsPurpose:

To learn the status of the project

Performed by:

Managers

Method: Practitioners report on

the status and plans of their projects, following specified formats and reporting on specific metrics




All Rights Reserved

CSE7315M37

Reviews (continued)Typical Duration: A few hours to several daysAchieves: Uncovers problems (or, at least, symptoms)Drawbacks: Does not identify solutions May motivate hiding of problemsAdvantages: Relatively inexpensive Tends to get everyone back on the same

track




All Rights Reserved

CSE7315M37

AuditsPurpose: To study a project in detail and find problems. Also to keep things on trackPerformed by: Independent technical experts, often outsidersMethod: Experts question practitioners and examine artifacts

of their process to determine what is happeningTypical Duration: Several days to several weeks




All Rights Reserved

CSE7315M37

Audits (continued)

Achieves:

Uncovers real problems more often

Informs staff that management cares about the results

Advantages:

Tends to uncover real problems

Tends to confirm or disprove suspected problems




All Rights Reserved

CSE7315M37

Audits (continued)

Drawbacks: More expensive than a review Does not identify solutions May motivate hiding of problems Can generate hostility and lack of cooperation Can de-motivate Outsiders often do not understand and even if they do, they are not believed

You don’ttrust us!




All Rights Reserved

CSE7315M37

Capability EvaluationsPurpose: To determine if an organization is capable of doing a

job wellPerformed by: Outside experts in all affected areasMethod: -- Many methods have been used for years -- For example, each of the armed services has ways

of doing contractor capability evaluations before selecting contractors

-- SEI has defined the SCE method, based on the CMM




All Rights Reserved

CSE7315M37

SEI Definition

Typical duration: several days

A software capability evaluation is an appraisal by a trained team of

professionals to identify contractors who are qualified to perform the software work or to monitor the

state of the software process used on an existing software effort.




All Rights Reserved

CSE7315M37

Capability Evaluations (continued)

• The SEI SCE (Software Capability Evaluation) is like an organized audit where the CMM is the model against which the program is evaluated

CMM 1.1




All Rights Reserved

CSE7315M37

SEI Capability Evaluations What They Achieve

• They are based on well established best practices for software development

• So the results can be seen in comparison with accepted best practices

• They uncover problems (or, at least, symptoms)

• They note lack of problems• They note capabilities




All Rights Reserved

CSE7315M37

SEI Capability Evaluations Drawbacks

• They do not identify solutions• They may motivate hiding of

problems• They are relatively expensive• Inexperienced evaluators can

misinterpret the findings, resulting in incorrect or unfair results– Example: focusing on practices rather

than goals and commitments




All Rights Reserved

CSE7315M37

SEI Capability Evaluations Drawbacks (continued)

• Inexperienced or immature managers may not understand the conclusions, interpret them correctly, or act appropriately– Example: instituting metrics when the

problem is communication




All Rights Reserved

CSE7315M37

“Self” AssessmentsPurpose: To help an organization learn where it is, how to

improve, and what to do nextPerformed by: A team of inside and/or outside technical and management expertsMethod: There are many methods of self assessment SEI has defined the two methods of software capability

assessment based on the CMM There have been several versions of the SEI method,

which can cause confusion




All Rights Reserved

CSE7315M37

SEI Definition (from Paulk)

“A software process assessment is an appraisal by a trained team of

software professionals to determine the state of an organization's current software process, to determine the

high-priority software process-related issues facing an organization, and to obtain the organizational support for

software process improvement.”




All Rights Reserved

CSE7315M37

“Self” Assessments Achieve

• They uncover problems • They identify recommended

solutions• They an be used to motivate– People “buy in” when they participate

in the assessment




All Rights Reserved

CSE7315M37

“Self” Assessment Drawbacks

• Can mislead if too many internal participants

• Can be misused as a sort of audit or proof of capability

• Can be very expensive• Can do more harm than good if

practitioners are motivated to change and management does not support the changes




All Rights Reserved

CSE7315M37

Assessment vs. Evaluation

CMM 1.0

Evaluations

• Find Problems

• Evaluate Capability

• For External Use

Assessments

• Find Problems

• Find Solutions

• Motivate Improvement

• For Internal Use

CMM 1.1

CMM 1.0

Humphrey5-LevelModel




All Rights Reserved

CSE7315M37

SEI Assessment Methods

• SPA– Software Process Assessment

• CBA-IPI– CMM Based Appraisal for Internal

Process Improvement

With both methods, SEI receives results and records confidentially. Results are published in aggregate form, quarterly.




All Rights Reserved

CSE7315M37

SEI Publishes a Maturity Profile

http://www.sei.cmu.edu/sema/profile.html

64.8

26.7

8.5

0

10

20

30

40

50

60

70

Percent of Reports

Commercial

DefenseContractor

Military




All Rights Reserved

CSE7315M37

Maturity Profile as ofMarch, 2001

32.2

39.3

19.4

5.4 3.7

0

5

10

15

20

25

30

35

40

1 2 3 4 5

Percent of Total




All Rights Reserved

CSE7315M37

SPA Method• Version 1 was based on a questionnaire or

survey, derived from Watts Humphrey’s model• Typical duration: 1 week• Several intermediate versions have been

based on intermediate versions of the CMM• The most recent and current version is based

on CMM 1.1• In each case there have also been several

degrees of “certification” of assessors, leading to many misunderstandings and unverifiable claims




All Rights Reserved

CSE7315M37

Training for SEI Assessments • Initial Assessment Method (1988-93)– Assessors were trained by SEI– Later, assessors were given “certification”

by SEI– An assessment was deemed “more

reliable” if one or more trained and/or certified assessors were involved

– See Humphrey, chapter 3, for more details• Later Assessment Methods– Individuals and/or organizations certified as

assessment team leaders and/or as trainers




All Rights Reserved

CSE7315M37

Confidentiality

• SEI does not release results of any specific assessment or confirm or deny the validity of any assessment

• They only release composite results to show overall industry characteristics and trends




All Rights Reserved

CSE7315M37

CBA-IPI Method• Typical duration: 1-2 weeks• More comprehensive than SPA• Places increased emphasis on CMM, when

compared with the SPA method• Intended to foster improvement for an

organization, not just focused on software• May be conducted by internal or external

personnel• Must be “led” by an “SEI Certified Lead Appraiser”




All Rights Reserved

CSE7315M37

CMM-based Appraisals

Maturity LevelsMaturity Levels

Key Process AreasKey Process Areas

Common FeaturesCommon Features

Key PracticesKey Practices

Goals

Implementation

Infrastructureor Activities

ProcessCapability

Address

Describe

Achieve

Contain

Organized by

Contain

Indicate

Questions




All Rights Reserved

CSE7315M37

SPA and CBA-IPI IssuesThese appraisals are intended to help an

organization evaluate itself– But too often, organizations will

publicize their appraisal results in the form of an achieved SEI maturity level

– SEI states that “when the focus is primarily on achieving a maturity level, it can distort the purpose of process improvement by diverting attention from genuine process improvement activities”




All Rights Reserved

CSE7315M37

SPA and CBA-IPI Issues(continued)

• Contracting agencies, especially in the government, may encourage incorrect use of assessments by insisting on specific SEI maturity levels for contract eligibility

• SEI specifically states that “finding a maturity level is optional” for SPA and CBA-IPI and “not intended or appropriate” for an SCE




All Rights Reserved

CSE7315M37

More SPA and CBA-IPI IssuesAssessment expense has generated many

lesser approaches– Typical appraisal cost: $80,000 (large org)• Assessment team (external): $45,000• Preparation $15,000• Time for participants $20,000

– Alternatives are OK for individual use, but the less extensive the less reliable the results

– Lack of external participation, while saving money, can also result in too-harsh or too-lenient appraisals

CSE 7315 - SW Project Management / Module 37 - Process Appraisal and AssessmentCopyright © 1995-2004, Dennis J. Frailey, All Rights Reserved

Slide 35

CSE7315M37

January 10, 2004

Low Cost Assessment Approaches




All Rights Reserved

CSE7315M37

“Temperature Taker”

• Select one or two random questions pertaining to key practices of a given KPA (a total of 6-12 questions for level 2, for example)

• Ask these questions of each project and discuss answers with participants

• Tally results at organization level to find strengths and weaknesses




All Rights Reserved

CSE7315M37

Characteristics of“Temperature Taker”

• Costs about 2 hours per project to collect data plus a few hours to tally

• Spots significant issues• And communicates the

organization’s concern about maturity

• But may miss significant issues




All Rights Reserved

CSE7315M37

“Continuous Record”

• Organization selects key practices to focus on within each KPA

• Each project maintains a list of key practices and a chart of whether they do each practice

• Display a graph in the work area and report monthly to the organization

• Tally at the organization level




All Rights Reserved

CSE7315M37

Characteristics of“Continuous Record”

• Costs less than 1 hour per project per reporting period

• Spots issues and trends• And communicates the

organization’s concern about maturity

• But relies on project self-assessment, so may still miss significant issues




All Rights Reserved

CSE7315M37

“Internal Assessment”

• Organization defines an assessment team of internal individuals

• Periodically, this team does an assessment

• Results are reported as in a full assessment




All Rights Reserved

CSE7315M37

Characteristics of“Internal Assessment”

• Costs less than a full assessment– No external participants– Assessors have a smaller “learning curve”

• Spots issues and trends• And communicates the organization’s

concern about maturity• But relies on internal participants, so

may still miss significant issues or blow issues out of proportion




All Rights Reserved

CSE7315M37

Module Summary• Appraisals of various kinds use

the CMM or other models to evaluate an organization’s maturity

• The more effective appraisal methods identify solutions as well as problems and foster buy-in and acceptance at all levels in the organization




All Rights Reserved

CSE7315M37

END OFMODULE 37

Documents

CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved CSE7315M37