Early Adoption Experience with CMMI-ACQ · Early Adoption Experiences with CMMI for Acquisition 4 081508 Some definitions: Acquisition - “The process of obtaining products or services

Early Adoption Experiences with CMMI for Acquisition 081508

Sponsored by the U.S. Department of Defense© 2008 by Carnegie Mellon University

This material is approved for public release.

Early Adoption Experiences with CMMI® for Acquisition (CMMI-ACQ)

by Jon L Gross and Matt Fisher

SM CMM Integration, SCAMPI, SCAMPI Lead Appraiser, TSP, and IDEAL are service marks of Carnegie Mellon University.

® CMMI, Capability Maturity Model, CMM, and Carnegie Mellon are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. For more information on CMU/SEI Trademark use, please visit http://www-internal.sei.cmu.edu/teams/comms/trademarks-usage.html

http://www-internal.sei.cmu.edu/teams/comms/trademarks-usage.html

© 2008 Carnegie Mellon University

Early Adoption Experiences with CMMI for Acquisition

2081508

Outline of Presentation

• Process improvement needs in an acquisition organization

• High level summary of CMMI for Acquisition (CMMI-ACQ) and differences with CMMI for Development (CMMI-DEV)

• Description of early adoption experiences and risks identified• Teaching• Appraisals• Process Improvement Consulting

• Mitigation of early adoption risks



3081508



• High level summary of CMMI-ACQ and differences with CMMI- DEV





4081508

Some definitions:

Acquisition - “The process of obtaining products or services through a supplier agreement.” This would also include, outsourcing and/or supply chain; where supplier agreements are established.

Supplier agreement - “A documented agreement between the acquirer and supplier (e.g., contract, license, or memorandum of agreement).”

Source: CMMI-ACQ V1.2 model

Note: The laws and regulations of the country and organization are expected to be followed when applying CMMI-ACQ in improved processes.



5081508

What We Mean By Acquisition

AwardPhase

Pre-AwardPhase Post-Award Phase

RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals

• Acquisition planning• Bidders conference• RFP preparation• Solicitation

• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables

• Proposal evaluation• Best and final offers• Source selection

RFI TIMs

AwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMs

These are all potentials areas for acquisition work process improvement.

Acquisition Phases



6081508

Lifecycles Overlay – A complex environment

Lifecycle Milestones

• Product

• Acquisition

• Project

AwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMs

AwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMsAwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMs

AwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMs

AwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMs

AwardPhase


RFP

ContractAward

ContractCompletion

Initiation

Deliverables...

...

Proposals




RFI TIMs

0 1 2 N N+1

These all potentials areas for acquisition work process improvement.



7081508

A Survey of Software Projects

1994

1996

1998

2000

2002

2004

2006

0% 40% 60% 80% 100%20%

Software projects completed on time (Success) Projects canceled before completion (Failed)Late and over budget (Challenged)

16%

27%

26%

28%

31%31%

40%40%

28%28%

23%23%

53%

33%

46%

49%

29% 18%18% 53%

35% 19%19% 46%

34%

Ref: CHAOS Study, Standish Group, 1994 - 2006

Average cost growth exceeds 89%Average final product contains 61% of the originally specified features

15%15% 51%

What role did acquisition organization play in results? We do not know, not measured.



8081508

The State of Acquisition Practice

Is There an Acquisition Crisis?Investigation of one acquisition program showed:—System complexity and the program’s lack of experience in

procuring major systems caused serious cost growth.—Program lacked systems engineering and program management

expertise.—Requirements stabilization process was absent.—Program management does not enforce timely milestones,

timelines, and deliverables.—Program’s lack of process control made assessment of technical

risk impossible.—Program’s lack of short- and long-term budget tracking made cost

assessment nearly impossible.—Program did not manage risk.

Source: GAO Report



9081508

Supplier

Mismatch

Mismatch

mature acquirer mentors low maturity supplier

less mature acquirer derails mature supplier; encourages short cuts

Matchedacquirer and supplier are both high maturity

Disasterno disciplineno process

Technical & Management Skill

Low

Low

Hig

h

High

outcome not predictable

highest probability of success

no product supplier compromises processes

Acquirer/Supplier Mismatch

Acq

uire

r



10081508

Addressing Mismatch - Why Look in the Mirror?

“He who has the gold makes the rules” and pays the price.”

• All acquisitions begins inside the organization needing a development or an outsourcing.

• Acquisition work processes lead to supplier agreements.

• Insanity -- “… doing the same thing over and over and expecting different results. "-Benjamin Franklin

• Agencies have spent decades using magnifying glasses to look at the performance of the suppliers/contractors.

• We still have problems with program success.

It is time to start looking in a mirror



11081508

Acquisition Management

Acquisition activities deal with:

Acquisition project team

Supplier activities deal with:

Supplier project team

The overall project performance, schedule, cost

The Supplier’s performance, schedule, cost

WHO IS IN CHARGE?And who is eventuality accountable



12081508

• Based on the premise that “The quality of a system or product is highly influenced by the quality of the process used to develop or maintain it.” (CMMI-ACQ V1.2)

We could improve the process and practices of the Supplier•But the developers have a head start (CMMI-DEV based improvements are widespread) •Has not solved all the problems

• We could improve the processes and practices of the Acquirer by: •increasing the visibility of the Acquirer’s contribution to program success •defining, implementing, measuring and evolving effective acquisition processes and practices

What Can Be Done?



13081508

As the Acquirer, how would you like to work?

Processes can make the difference.



14081508

Ineffective Approach to Quality Acquisition

Historically, many acquisition organizations have approached quality by performing product (contractor or supplier) measurement only. But, this approach does not improve quality nor guarantee quality. It’s a strategy of defect detection, not defect prevention.

Measurement of final product attributes is too late ...the quality, good or bad, is already in the product.

100% inspection is only 80% effective — Dr. Joseph Juran

Acquisition Processes

Contractor/ Supplier Products and Processes

Examining with a ‘magnifying glass’



15081508

Expected Acquisition Organization Result?

As you move up in maturity level, project cost will decrease and distribution will become tighter and more predictable

Most acquisition organization do not measure their work processes.Most acquisition organizations tend to be optimistic in the program projections.

Benefits of measuring:• Better estimation• Less deviation from

targetNotional for acquisition processes.



16081508



• High level summary of CMMI-ACQ and differences with CMMI-DEV

• Description of early adoption experiences and risks identified

— Teaching

— Appraisals

— Process Improvement Consulting




17081508

CMMI V1.1March 2002

CMMI-AMV1.1

May 2005

CMMI-DEV V1.2August 2006

CMMI-ACQV1.2

Nov 2007

Evolution of CMMI Models*

CMMI-SVCTBD

Adapting CMMI for Acq

Jun 2006

SW-CMM

V1.11993

*Notional Time Scale

EIA 7311998

Guidance For Acquisition Improvement Has Evolved For Over a Decade.

SA-CMMV1.011996

SA-CMMV1.021998

SA-CMMV1.032002



18081508

Traditional Acquisition Activities

Acquisition Organization

The Supplier Agreement or Contract

SupplyingOrganization



19081508

Comparisons of CMMI-ACQ, V1.2 and CMMI-DEV, V1.2

CMMI-ACQ CMMI-DEV

For the acquirer (i.e., those who acquire, procure, or otherwise select and purchase products and services for business purposes, or those who outsource development and support)

For the product and service developer (i.e., those who develop or maintain products and services for business purposes)

Focus on the acquisition of products and services Focus on the development and maintenance of products and services

Generic practices are covered only in the Generic Goals and Generic Practices section

Generic practices are covered both in the Generic Goals and Generic Practices section and at the end of each process area

Explicit coverage of services Implicit coverage of services through the definition of the term “product,” which covers both products and services

Contains an Acquisition process area category, but no Engineering category

Contains an Engineering process area category, but no Acquisition category

Stages Acquisition Requirements Development at maturity level 2

Stages Requirements Development at maturity level 3

Categorizes Requirements Management as a Project Management process area

Categorizes Requirements Management as an Engineering process area

Contains typical work products and typical supplier deliverables

Contains typical work products

http://www.sei.cmu.edu/cmmi/models/ACQ-v12-comparetoDEV.html



20081508

ProjectManagement

Process AreasCategory

<Acquisition Requirements Development><Acquisition Technical Management><Acquisition Verification><Acquisition Validation><Agreement Management><Solicitation and Supplier Agreement Development>

Acquisition

Configuration ManagementProcess and Product Quality AssuranceMeasurement and AnalysisDecision Analysis and Resolution Causal Analysis and Resolution

Support

<Project Planning><Project Monitoring and Control><Integrated Project Management>Requirements ManagementRisk ManagementQuantitative Project Management

Organizational Process Focus<Organizational Process Definition>Organizational TrainingOrganizational Process PerformanceOrganizational Innovation and Deployment

ProcessManagement This chart shows all PAs

in CMMI-ACQ. All PAs in the Acquisition Category are unique to CMMI-ACQ.

The other PAs with blue and surrounded by <> contain additional specific practices that are only in the CMMI-ACQ model.

Requirements Management is in the Project Management category in CMMI-ACQ.

http://www.sei.cmu.edu/cmmi/models/ACQ-v12-comparetoDEV.html

CMMI-ACQ V1.2 Continuous Representation: PAs by Category



21081508

Supplier

Mismatch

Mismatch

mature acquirer mentors low maturity supplier

less mature acquirer derails mature supplier; encourages short cuts

Matchedacquirer and supplier are both high maturity

Disasterno disciplineno process

Technical & Management Skill

Low

Low

Hig

h

High

outcome not predictable

highest probability of success

no product supplier compromises processes

Acquirer/Supplier Mismatch

Acq

uire

r



22081508

Acquirer is in Charge

Acquisition Planning

RFP Prep.

Solicita- tion

Source Selection

System Acceptance

Program Leadership Insight / Oversight Transition

Plan Design Integrate & TestDevelop Deliver

Supplier Processes

CMMI for Acquisition

Operational and Business Need

Developer or Supplier Processes(CMMI for Development based process improvement.)

Acquirer Processes

Operational and Business Need Satisfied

Task Orders Do This After Basic Contract Awarded



23081508

Today’s Approach to Quality Acquisition

• A strategy of problem prevention: focus on the acquisition work processes.

• Fixing the “acquisition processes” lowers the probability bad products or services will be produced in the future acquisitions and supplier agreements.

Contractor/ Supplier Products and Processes

Acquisition Work Processes

Examining self with a

‘mirror’

The Supplier Agreement or Contract



24081508



• High level summary of CMMI-Acquisition and differences with CMMI-DEV





25081508

Early Experiences from Teaching - 1

Learning to Look in a Mirror is Tough

• “It is the suppliers problem, that is why we hired them.”

A few acquisition organizations are willing to focus on their internal improvement.• Some US Government Agencies• Some Foreign Government Agencies• Some commercial companies



26081508

Early Experiences from Teaching - 2

• They are using experiences from CMMI-DEV process improvement and applying them to acquisition work processes.

• Still holding the supplier accountable for contract performance.



27081508

Early CMMI-ACQ Appraisal Experiences - 1

General Motors performed SCAMPI A, B and C Appraisals, using process areas from CMMI-DEV and the Adapting CMMI for Acquisition Organizations.• SCAMPI Method and Classes work, without change• Depends on acquisition organization defining their work

processes

Gaining continuing Senior Level Sponsorship is Key, but can be politically sensitive.• External pressure can be essential (e.g., GAO, Congress,

CEO)• Strategic View is important “I need to improve”

A few US Government acquisition organizations are doing model based process improvement.



28081508


The SCAMPI Appraisal Family of methods works for CMMI-ACQ, in the same way it works for CMMI-DEV.

SCAMPI C: provides a wide range of options, including characterization of planned approaches to process implementation according to a scale defined by the user

SCAMPI B: provides options in model scope and organizational scope, but characterization of practices is fixed to one scale and is performed on implemented practices

SCAMPI A: Is the most rigorous method, and is the only method that can result in ratings.

breadth of tailoring

C

A

B depth of investigation

SCAMPI Lead Appraiser must be authorized for CMMI-ACQ.

SCAMPI Family



29081508


SCAMPI Appraisal Approach for CMMI-ACQ:

• Assure the appraisal team members have experience and understanding of acquisition work.

• Assure model understanding by the organizational unit in preparing for appraisal.

— SCAMPI Class B and C appraisals are the best starting point— Uncovers model issues in a less intense appraisal environment— Allows time to align work processes with appraisal objective

evidence expectations.

• Appraisal Team Members need to complete SEI Authorized Introduction to CMMI-ACQ V1.2 Training.



30081508

Early CMMI-ACQ Process Consulting Experiences• Transition from current process improvement model to CMMI-ACQ

• One Government organization is developing a transition process, with entry and exit criteria

• A commercial company is transitioning from acquisition processes based on CMMI-DEV to new processes based on CMMI-ACQ

• Some organizations are starting with blank sheets of paper

• Lack of knowledge of model detail by senior management

• Sponsors often do not have enough detailed knowledge of the models to understand the significance, benefits, or effort required

• Budget and/or senior management changes often affect process improvement programs first



31081508



• High level summary of CMMI-ACQ and differences with CMMI-DEV

• Description of early adoption experiences and risks identified

— Teaching

— Appraisals

— Process Improvement Consulting




32081508

The Bottom Line or Mission Must Drive ImprovementImprovement means different things to different organizations:

• What are your mission or business goals?

• How do you measure progress?

Improvement is a long-term, strategic effort:

• What is the expected impact on the mission or bottom line?

• How will impact be measured?



33081508

Getting Started: Key Success Factors for an Effective PI Effort

Before major change can even be contemplated, top management must recognize the need and be sold on the idea.

The Must Haves:

1. A compelling reason for change

2. Leadership of the change effort by the top executive in the organization — responsibility cannot be delegated

3. Informed commitment of the top management team

4. Designation of a primary PI Leader and an adequate mandate for change

5. Sound performance measures that drive change

Source: Mastering Process Improvement Course



34081508

Getting There Means CHANGING!Acquisition organizations have many barriers to address to reach the culture of measured process performance.

Improvement of acquisition work processes is still in the “Innovator” or “Earlier Adopter” stage of transition.



35081508

Why So Long for Process Improvement?

Magnitude of Technological Change Sought

Years Months Weeks Small Large

Time to Adjust

Skills

Procedures

Structure

Strategy

Culture

Levels of Learning Required

Diagram adapted from Adler

Most Acquisition Organizations are Large.

Do you have long term sponsorship?



36081508

Organizing Process Improvement

InitiateInitiate DiagnoseDiagnose EstablishApproachEstablishApproach ActAct LearnLearn

Get Started Where Am I? What ShouldI Do?

Take Action Learn fromExperience

Re-assessed as needed for further improvement

Shows benefit of process improvement

Helps get management buy-in

CMMI-ACQ appraisal baselines acquisition work processes

Provides an agenda for improvement

Capability levels in Process Areas help set priorities and goals

Define Process Areas and characteristics of desirable practices

Provides a framework to allow fine- tuning to improvement process

Reference: IDEAL



37081508

Ongoing Communication is Key to Adoption

How individuals and organizations commit to change

Time

ContactAwareness

Understanding

Adoption

Institutionalization

Com

mitm

ent

Installation

Daryl R. Conner and Robert W. Patterson. “Building Commitment to Organizational Change,”Training and Development Journal (April 1983):18-30.

Acquisition organizations

are just starting the adoption

process.



38081508

Applying the CMMI-ACQ

Use the same basic steps:

• CMMI-ACQ terminology must be mapped onto local acquisition organization terminology and the local way of doing business.

• The generic organization terms (e.g., project manager, project, senior manager) of the CMMI-ACQ must be mapped onto the local organization’s management structure.

• The CMMI-ACQ should be applied using intelligence, common sense, and professional judgment. • The CMMI-ACQ is not carved in stone.



39081508

Process Improvement Consulting

It is all about:

• getting and continuing long term sponsorship

• a willingness and courage to look internally

• a willingness to establish documented processes

• establishing an internal process improvement support effort



40081508

Conclusion

CMMI-ACQ provides the opportunity to do the following:

• Improve acquisition program results, by improving operating processes and practices in acquisition organizations

• Improve the management and technical work skills— acquirer-supplier— acquirer-acquirer— acquirer-user

• Encourage strategic acquirer-supplier teamwork• Extend enterprise-wide process improvement



41081508

Contact Information

Mr. Jon L. GrossSoftware Engineering InstituteCarnegie Mellon University4301 Wilson Blvd, Suite 200Arlington, VA 22203-1867USAOffice: (703) 908-8238Email: [email protected]

Dr. Matthew FisherSoftware Engineering InstituteCarnegie Mellon University4500 Fifth AvePittsburgh, PA 15213-3890USAOffice: (412) 268-5877Email: [email protected]



42081508

BACK-UP SLIDES



43081508

CMMI-ACQ v1.2 Acquisition Category Process Areas

Acquisition Requirements Development

Solicitation & Supplier

Agreement Development

Agreement Management

Acquisition Technical

Management

Acquisition Validation

Acquisition Verification

CMMI Framework

16 Project, Organizational,

and Support Process Areas



44081508

Why Is a Process Model Important?

A process model (CMMI-ACQ) provides

• a place to start

• the benefit of a community’s prior experiences

• a common language and a shared vision

• a framework for prioritizing actions

• a way to define what “improvement” means for your organization

Remember: “All models are wrong; some are useful.” - George Box

Documents

Early Adoption Experience with CMMI-ACQ · Early Adoption Experiences with CMMI for Acquisition 4 081508 Some definitions: Acquisition - “The process of obtaining products or services