LASPIN-INCOSE – 7/30/031 LASPIN-INCOSE Meeting Los Angeles, CA July 30, 2003 Ricardo Valerdi The Aerospace Corporation & University of Southern California

1

USC

C S E University of Southern CaliforniaCenter for Software Engineering

LASPIN-INCOSE – 7/30/03

LASPIN-INCOSE MeetingLos Angeles, CA

July 30, 2003

Ricardo ValerdiThe Aerospace Corporation &University of Southern CaliforniaCenter for Software Engineering

2

USC



Outline• Goals of this talk

• Background, key ideas, and definitions

• Overview of COSYSMO

• Delphi results

• Estimation example

• Data collection process

• Demo

• Next steps

3

USC



Goals of this talk

1. Provide an overview of COSYSMO

2. Demonstrate the prototype

3. Share future plans for data collection

4. Get feedback/suggestions on approach and data collection

4

USC



“All models are wrong, but some of them

are useful”

- W. E. Deming

Source: www.deming.org

5

USC



Key Definitions & ConceptsCalibration: the tuning of parameters based on project

data CER: a model that represents the cost estimating

relationships of factorsCost Estimation: prediction of both the

person-effort and elapsed time of a projectDriver: A factor that is highly correlated to the amount of

Systems Engineering effortParametric: an equation or model that is approximated by

a set of parametersRating Scale: a range of values and definitions for a

particular driverUnderstanding: an individual’s subjective judgment of

their level of comprehension

6

USC



COCOMO II• COCOMO is the most widely used, thoroughly

documented and calibrated software cost model• COCOMO - the “COnstructive COst MOdel”

– COCOMO II is the update to COCOMO 1981– ongoing research with annual calibrations made

available • Originally developed by Dr. Barry Boehm and

published in 1981 book Software Engineering Economics

• COCOMO II described in Software Cost Estimation with COCOMO II (Prentice Hall 2000)

7

USC



USC Center for Software Engineering (CSE)• Researches, teaches, and practices CMMI-based Software

engineering

– Systems and software engineering fully integrated • Collaborative efforts between Computer Science (CS) and

Industrial Systems Engineering (ISE) Departments• COCOMO Suite of models

– Cost, schedule: COCOMO II, CORADMO, COCOTS– Quality: COQUALMO– Systems Engineering: COSYSMO

• Applies and extends research on major programs (DARPA/Army, FCS, FAA ERAM, NASA Missions)

• Uses mature 7-step model development methodology

8

USC



• Marilee Wheaton, Aerospace/INCOSE Corporate Advisory Board• Dave Hickman, Aerospace/INCOSE Corporate Advisory Board• Dr. Barry Boehm, USC/INCOSE Fellow• Dr. Elliot Axelband, USC/INCOSE• Don Greenlee, SAIC/INCOSE V&V• Eric Honour, INCOSE SECOE• Paul Robitaille, LMCO/INCOSE Corporate Advisory Board• Garry Roedler, LMCO/ISO-IEC 15288 • Chris Miller, SPC/INCOSE Measurement Working Group• Dr. John E. Rieff, Raytheon/INCOSE Corporate Advisory Board• Jim VanGaasbeek, Northrop/INCOSE Corporate Advisory Board

INCOSE Involvement

9

USC



• Commercial Industry (15)– Daimler Chrysler, Freshwater Partners, Galorath, Group

Systems.Com, Hughes, IBM, Cost Xpert Group, Microsoft, Motorola, Price Systems, Rational, Reuters Consulting, Sun, Telcordia, Xerox

• Aerospace Industry (6)– BAE, Boeing, Lockheed Martin, Northrop Grumman,

Raytheon, SAIC

• Government (8)– DARPA, DISA, FAA, NASA-Ames, NSF, OSD/ARA/SIS,

US Army Research Labs, US Army TACOM

• FFRDC’s and Consortia (4)– Aerospace, JPL, SEI, SPC

• International (1)– Chung-Ang U. (Korea)

USC-CSE Affiliates (34)

*COSYSMO Contributors

10

USC



7-step Modeling MethodologyAnalyze Existingliterature

1

2

3

4

5

6

7

PerformBehavioral Analysis

Identify RelativeSignificance

Perform Expert-Judgement, DelphiAssessment

Gather Project Data

Determine BayesianA-Posteriori Update

Gather more data;refine modelDetermine statistical significance

11

USC



COCOMO II• Software• Development phases• 20+ years old• 200+ calibration points• 23 Drivers• Variable granularity• 3 anchor points• Size is driven by SLOC

COSYSMO• Systems Engineering• Entire Life Cycle• 2 years old• ~3 calibration points• 18 drivers• Fixed granularity• No anchor points• Size is driven by

requirements, I/F, etc.

Model Differences

12

USC



COSYSMO: Overview• Parametric model to estimate system

engineering costs• Includes 4 size & 14 cost drivers• Covers full system engineering lifecycle• Focused on use for Investment Analysis,

Concept Definition phases estimation, tradeoff & risk analyses– Input parameters can be determined in

early phases

13

USC



COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

+Volatility Factor

- Application factors-8 factors

- Team factors-6 factors

- Schedule driver WBS guided by ISO/IEC 15288

COSYSMO Operational Concept

14

USC



EIA/ANSI 632

EIA/ANSI 632 - Provide an integrated set of fundamental processes to aid a developer in the engineering or re-engineering of a system

Breadth and Depth of Key SE StandardsSystem life

ISO/IEC 15288

Level of

deta

il

Conceptualize DevelopTransition to

Operation

Operate,Maintain,

or EnhanceReplace

or Dismantle

Processdescription

High levelpractices

Detailedpractices

ISO/IEC 15288 - Establish a common framework for describing the life cycle of systems

Purpose of the Standards:Purpose of the Standards:

IEEE 1

220

IEEE 1220 - Provide a standard for managing systems engineeringSource : Draft Report ISO Study Group May 2, 2000

15

USC



Data will drive the Evolution Path & Scope of the Model

Oper Test & Eval

1. COSYSMO-IP

2. COSYSMO-C4ISR

3. COSYSMO-Machine

4. COSYSMO-SoS

Global Command and Control System

Satellite Ground Station

Joint Strike Fighter

Future Combat Systems

DevelopConceptualizeTransition to Operation

Operate, Maintain, or Enhance

Replace or Dismantle

16

USC



COCOMO-based Parametric Cost Estimating Relationship

Where: PMNS = effort in Person Months (Nominal Schedule)

A = constant derived from historical project data Size = determined by computing the weighted average of the size drivers E = exponent representing the economy/diseconomy of scale dependent on size drivers (4), currently set to 1n = number of cost drivers (14)EM = effort multiplier for the ith cost driver. The geometric product results

in an overall effort adjustment factor to the nominal effort.

i

n

i

ENS EMSizeAPM

1)(

17

USC



4 Size Drivers1. Number of System Requirements

2. Number of Major Interfaces

3. Number of Operational Scenarios

4. Number of Critical Algorithms

• Each weighted by complexity, volatility, and degree of reuse

18

USC



Number of System RequirementsThis driver represents the number of requirements for the system-of-interest at a specific level of design. Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor. System requirements can typically be quantified by counting the number of applicable “shall’s” or “will’s” in the system or marketing specification. Do not include a requirements expansion ratio – only provide a count for the requirements of the system-of-interest as defined by the system or marketing specification.

Easy Nominal Difficult

- Well specified - Loosely specified - Poorly specified

- Traceable to source - Can be traced to source with some effort

- Hard to trace to source

- Simple to understand - Takes some effort to understand - Hard to understand

- Little requirements overlap - Some overlap - High degree of requirements overlap

- Familiar - Generally familiar - Unfamiliar

- Good understanding of what’s needed to satisfy and verify requirements

- General understanding of what’s needed to satisfy and verify requirements

- Poor understanding of what’s needed to satisfy and verify requirements

19

USC



14 Cost Drivers

1. Requirements understanding

2. Architecture complexity

3. Level of service requirements

4. Migration complexity

5. Technology Maturity

6. Documentation Match to Life Cycle Needs

7. # and Diversity of Installations/Platforms

8. # of Recursive Levels in the Design

Application Factors (8)

20

USC



Requirements understanding This cost driver rates the level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc.

Very low Low Nominal High Very High

Poor, unprecedented system

Minimal, many undefined areas

Reasonable, some undefined areas

Strong, few undefined areas

Full understanding of requirements, familiar system

21

USC



14 Cost Drivers (cont.)

1. Stakeholder team cohesion

2. Personnel/team capability

3. Personnel experience/continuity

4. Process maturity

5. Multisite coordination

6. Tool support

Team Factors (6)

22

USC



Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team.

Very Low Low Nominal High Very High

Culture Stakeholders with diverse expertise, task nature, language, culture, infrastructure Highly heterogeneous stakeholder communities

Heterogeneous stakeholder communitySome similarities in language and culture

Shared project culture

Strong team cohesion and project cultureMultiple similarities in language and expertise

Virtually homogeneous stakeholder communitiesInstitutionalized project culture

Communication Diverse organizational objectives

Converging organizational objectives

Common shared organizational objectives

Clear roles & responsibilities

High stakeholder trust level

23

USC



Additional Proposed Drivers• # and diversity of installations/platforms phased out • # of years in operational life cycle• Quality Attributes• Manufacturability/Producibility• Degree of Distribution

24

USC



Delphi Results• From last COSYSMO Working

Group Meeting in Keystone, CO• 2 rounds• Avg. increased for all drivers

except TOOL• About 1/3 of the responses

were adjusted as a result of the discussion

• Goal is to get 30+ Delphi surveys filled out by October

Gary Hafen, LMCOBarry Boehm, USCDon Reifer, USCPaul Mohlman, AeroRob Flowe, DAURick Edison, LMCOGary Thomas, RaytheonGarry Roedler, LMCOCarl Newman, LMCO

25

USC



# of Req

0.00

5.00

10.00

15.00

20.00

25.00

1

26

USC



Req Und

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

1

27

USC



Team

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

1

28

USC



Estimation ExampleYou are the SE working on an upgrade of a legacy system with a pretty good understanding of requirements (“Nominal” rating)…







1.0

Requirements Understanding rating scale:

0.9 0.811.21.4

You estimate that the job will require 12 Person Months. This driver will have no additional impact on the cost of the job (effort is multiplied by 1.0).

29

USC



Estimation Example (cont)…all of a sudden…the customer adds a requirement to make the new system backwards compatible with the old one.

Your overall understanding of the requirements has decreased, your schedule and resources are fixed, and the amount of work has increased.

Mayday!

30

USC



Estimation Example (cont)Justify to Program Manager/Director your request to increase resources in this area.







1.0

Requirements Understanding rating scale:

0.9 0.811.21.4

The effort is multiplied by 1.2!Instead of 12 PM, this job will require 14.4 PM of Systems Engineering work (additional $19.2k*)

*assumes $8k/PM

31

USC



Data Collection Process• Project & people are identified

• Systems engineer• Cost estimator/data base manager

• Job/task codes in accounting system are mapped to COSYSMO• Meta data is collected

• System scope• Life cycle• Application domain

• Cost drivers are rated• Interaction between SE, Cost, USC

• Data is entered into secure repository at USC

32

USC



Safeguarding Procedures• Data identification

– Only affiliate & Dr. Boehm know the OID (XXX) and only affiliate knows PID (YYY)

• Data storage– Stand-alone computer at USC with one-way

access to the network– In a room with cypher lock & limited access

• Data access– Non-disclosure agreements signed– Controlled access to data by researchers (US

Citizens only)

33

USC



USC/Raytheon myCOSYSMO* Demo

*Developed by Gary Thomas

at Raytheon Garland

34

USC



Parametric Cost Model Critical PathUsual # Months*

6 Converge on cost drivers, WBS

6 Converge on detailed definitions and rating scales

12 Obtain initial exploratory dataset (5-10 projects)

6 Refine model based on data collection & analysis experience

12+ Obtain IOC calibration dataset (30 projects)

9 Refine IOC model and tool

Critical Path Task

*Can be shortened and selectively overlapped

35

USC



Calendar of Activities: 2003

2003 2004

INCOSE 2003(Washington, DC)

USC CSE Annual Research Review(Los Angeles, CA)

INCOSE IW(Tampa, FL)

COCOMO Forum(Los Angeles, CA)

Conference on Systems Integration(Hoboken, NJ)

J F M A M J J A S O N D

Practical Software & Systems Measurement Workshop (Keystone, CO)

Working Group Meeting

INCOSE/SCEA Meeting(Chantilly, VA)

36

USC



Don’t be left out!

37

USC



Questions or Comments?Ricardo Valerdi

[email protected]

Websites

http://sunset.usc.edu

http://valerdi.com/cosysmo

Books

Boehm, B., et al, Software Cost Estimation with COCOMOII, 1st Ed, Prentice Hall, 2000

Blanchard, B., Fabrycky, W., Systems Engineering and Analysis, 3rd Ed, Prentice Hall, 1998

Boehm, B., Software Engineering Economics, 1st Ed, Prentice Hall, 1981

Documents

LASPIN-INCOSE – 7/30/031 LASPIN-INCOSE Meeting Los Angeles, CA July 30, 2003 Ricardo Valerdi The Aerospace Corporation & University of Southern California