When Do You Need Systems of Systems Engineering: A Quantitative Analysis Jo Ann Lane 17 March 2009 University of Southern California Center for Systems

When Do You Need Systems of When Do You Need Systems of Systems Engineering: Systems Engineering: A Quantitative AnalysisA Quantitative Analysis

Jo Ann Lane17 March 2009

University of Southern CaliforniaCenter for Systems and Software Engineering

March 2009 USC CSSE Annual Research Review 2

OverviewOverview

• Key definitions

• Scope of research

• Methodology

• Model implementation

• Results of research

• Conclusions and future work


What is a “System of Systems”?What is a “System of Systems”?

• Very large systems developed by creating a framework or architecture to integrate constituent systems

• SoS constituent systems independently developed and managed– New or existing systems in various stages of development/evolution– May include a significant number of COTS products– Have their own purpose– Can dynamically come and go from SoS

• SoS exhibits emergent behavior not otherwise achievable by component systems

• Typical domains– Business: Enterprise-wide and cross-enterprise integration to support

core business enterprise operations across functional and geographical areas

– Military: Dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment

Based on Mark Maier’s SoS definition [Maier, 1998]


Types of SoSTypes of SoS• Virtual [Maier, 1998]

– Lacks a central management authority and a clear SoS purpose– Often ad hoc and may use a service-oriented architecture where

the constituent systems are not necessarily known

• Collaborative [Maier, 1998]– Constituent system engineering teams work together more or

less voluntarily to fulfill agreed upon central purposes – No SoSE team to guide or manage activities of constituent

systems

• Acknowledged [Dahmann, 2008]– Have recognized objectives, a designated manager, and

resources at the SoS level (SoSE team)– Constituent systems maintain their independent ownership,

objectives, funding, and development approaches

• Directed [Maier, 2008]– SoS centrally managed by a government, corporate, or Lead

System Integrator (LSI) and built to fulfill specific purposes– Constituent systems maintain ability to operate independently,

but evolution subordinated to centrally managed purpose

This research focused on identifying the “home-ground” for these two types of SoSs...


Scope of ResearchScope of Research

• Research question – When is it cost effective to establish and use a system of

systems engineering (SoSE) team to oversee and guide the evolution of a system of systems (SoS)?

• Hypothesis – There exists a threshold where it is more cost effective to

manage and engineer capability changes to an SoS using an SoSE team and this threshold can be determined by modeling the SoS system complexity and desired capability interdependency characteristics.

Focus is on software-intensive SoSs owned by the United States Department of Defense (DoD)...


Statement of Topic and Contribution Statement of Topic and Contribution (continued)(continued)

• Research contribution – Provides guidance to DoD leadership with respect to the

management of sets of inter-related systems that are functioning as a system of systems.

– Guidance also applies to SoSs in other domains that are managed as collaborative or acknowledged SoSs

– Model for management and engineering guidance also provides• A method for conducting trade-off analyses for different approaches

for implementing a given SoS capability for a given SoS

• A model that can evolve into an SoSE cost model through calibration for a given SoS or SoS domain

• A cost model that can better model complex systems


MethodologyMethodology• Using COSYSMO, developed a process model that can

compare the SoS management strategies as SoS characteristics are varied– SoS size (number of constituent systems)

– Size of SoS capability (number of equivalent nominal requirements)

– Scope of SoS capability (number of constituent systems affected by SoS capability)

– Constituent system volatility (level of constituent system change being engineered at the same time as SoS capability)

• Process model based on data from– 18 large-scale DoD SoS programs

– 16 DoD systems that participate as constituent systems in one or more SoSs

• Analyze model outputs to determine under what conditions an SoSE team is cost effective


SoSE Process Model OverviewSoSE Process Model Overview• Purpose

– Estimate and compare the effort required to implement an SoS capability using two different management approaches

• Collaborative (no SoSE team)• Acknowledged (SoSE with limited authority/control)

• Assumptions and constraints– All constituent systems currently exist and have their own evolutionary paths

based on system-level stakeholder needs/desires– Model assumes SoSE and traditional SE teams are using relatively mature

processes– SoS capabilities are software-intensive– No SoS capability/requirements volatility– SoS internal volatility represented by constituent system volatility– No accommodation of schedule factors or the asynchronous nature of SoS

constituent system upgrades– Management of SoS internal interfaces reduces complexity for systems


Systems Engineering Requirements CategoriesSystems Engineering Requirements Categories

• Requirements related to SoS capabilitiesa) Acknowledged SoS: Initially engineered at SoS level by SoSE

team with support from constituent system engineers for those systems impacted by the SoS capability, then allocated to constituent systems for further SE

b) Collaborative SoS: Not engineered at the SoS level, but must be engineered fully at the constituent system level through collaborative efforts with other constituent system engineers

• Non-SoS requirements related to constituent system stakeholder needs– Must be monitored by SoSE team to identify changes that might

adversely impact SoS– Represents on-going volatility at the constituent system level that

is occurring in parallel with SoS capability changes


SoSE Model StructureSoSE Model Structure

Focus is on software-intensive SoSs owned by the US DoD, the number and volatility of constituent systems within an SoS, and the complexity of typical capability enhancements to the SoS...


System Capability

Effort for a “collaborative”

SoS

Effort using an “acknowledged”

SoSE teamEquivalent set of

“sea-level” requirements

Conversion to COSYSMO size units

Calculations based on SoS characteristics/size and capability implementation approach using

COSYSMO algorithm

Overview of SoSE SDM FlowOverview of SoSE SDM Flow


Model Parameters by SDM ConstructModel Parameters by SDM Construct

• Stocks– Inputs

• SoS Equivalent Requirements– Outputs

• SoSE Effort• SoS Upgrade Effort with

SoSE• SoS Upgrade Effort without

SoSE

• Flows– Capability Rate– SoSE Effort Rate– SE Effort Rate with SoSE– SE Effort Rate without SoSE

• Converter Parameters– COSYSMO effort multipliers

• COSYSMO SoSE EM• COSYSMO SE EM with

SoSE• COSYSMO SE EM without

SoSE• COSYSMO SE EM

– SoS complexity factors• Number of systems in SoS• Number of systems affected

by capability• Average system rate of

change

General Form of COSYSMO EquationGeneral Form of COSYSMO Equation Effort (person months) = [38.55 * EM * (size)1.06] / 152


SoSE Effort MultiplierSoSE Effort Multiplier

2.50


Effort Multiplier for SoSE Monitoring of Effort Multiplier for SoSE Monitoring of Constituent System RequirementsConstituent System Requirements

0.47


SE Effort Multiplier for SoS Requirements SE Effort Multiplier for SoS Requirements withwith SoSE Support SoSE Support

1.06


SE Effort Multiplier SoS Requirements SE Effort Multiplier SoS Requirements withoutwithout SoSE Support SoSE Support

1.79


SE Effort Multiplier for System-Specific SE Effort Multiplier for System-Specific (Non-SoS) Requirements(Non-SoS) Requirements

0.72


Effort CalculationsEffort Calculations

SoSE Effort

SoSE Effort = 38.55*[((SoSCR/SoSTreq)*(SoSTreq)1.06 *EMSoS-CR)+ ((SoSMR/SoSTreq)*(SoSTreq)1.06 * EMSoS-MR)/152]

Where:

Total SoSE requirements = SoS Capability Requirements + SoS “Monitored” Requirements

SoS “monitored” reqs = [∑SE non-SoS requirements being addressed current upgrade cycles for all SoS constituent systems] * “Oversight Factor”

“Oversight Factor” = 5% , 10%, 15% (these values are based on expert judgment from various CSSE affiliates and the SoS SE Guidebook team)

Based on COCOMO II approach for combining components with different EMs (SoS changes and Constituent System oversight)


Effort Calculations Effort Calculations (continued)(continued)

Single System Effort with Support from SoSE Team

Total single system reqsw-SoSE = SoS requirements allocated to system + SE reqs in upgrade cycle

Single system SE Effort with SoSE Team

= 38.55*[1.15*( (SoSCSalloc / CSTreqSoSE)*( CSTreqSoSE)1.06* EMCS-CRwSOSE) +

(CSnonSoS / CSTreqSoSE)*( CSTreqSoSE)1.06* EMCSnonSOS] /152

Based on COCOMO II approach for combining components with different EMs plus including a 15% “tax” to support SoSE team in their engineering effort for the SoSE requirements. 15% represents half of the system design effort in the EIA 632 tasks.


Effort Calculations Effort Calculations (continued)(continued)

Single System Effort with No SoSE Team Support

Total single system reqs wo-SoSE = SoSE capability reqs + SE non-SoS requirements

Single system SE Effort without SoSE Team =

38.55*[(( SoSCR / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCS-CRnSOSE) +

((CSnonSoS / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCSnonSOS)] /152

Based on COCOMO II approach for combining components with different EMs (SoS changes and non-SoS changes)


Range of SoS Complexity Factor ValuesRange of SoS Complexity Factor Values

SoSE Model Parameter

Description Range of Values

SoS Size Number of constituent systems within the SoS

2-200

SoS Capability Size Number of equivalent nominal requirements as defined by COSYSMO

1-1000

Constituent System Volatility

Number of non-SoS changes being implemented in each constituent system in parallel with SoS capability changes

0-2000

Scope of SoS Capability

Number of constituent systems that must be changed to support capability

One to SoS Size (total number of constituents systems within the SoS)

SoSE Oversight Factor

Oversight adjustment factor to capture SoSE effort associated with monitoring constituent system non-SoS changes

5%, 10%, and 15%


Results of ResearchResults of ResearchScenario 1 (SoS Size Varies) Scenario 2 (SoS Size Varies)

Scenario 3 (SoS Size Varies) Scenario 4 (SoS Size Varies)

Relative Cost of Collaborative and Acknowledged SoSECapability Affects Half of the Systems

System Volatility = 100 Reqs and SoS Capability = 100 Reqs

-300.00

0.00

300.00

600.00

900.00

1200.00

1500.00

1800.00

0 50 100 150 200 250

Number of Systems

Sav

ing

s (P

erso

n M

on

ths)

OSF 5%

OSF 10%

OSF 15%



-200.00

0.00

200.00

400.00

600.00

800.00

0 50 100 150 200 250

Number of Systems

Sa

vin

gs

(P

ers

on

Mo

nth

s)

OSF 5%

OSF 10%

OSF 15%

Relative Cost of Collaborative and Acknowledged SoSECapability Affects One-Fourth of the Systems


-400.00

0.00

400.00

800.00

1200.00

1600.00

2000.00

0 50 100 150 200 250

Number of Systems

Sa

vin

gs

(Pe

rso

n M

on

ths

) OSF 5%

OSF 10%

OSF 15%



-200.00

-100.00

0.00

100.00

200.00

300.00

400.00

0 50 100 150 200 250

Number of Systems

Sa

vin

gs

(P

ers

on

Mo

nth

s)

OSF 5%

OSF 10%

OSF 15%


Results of Research Results of Research (continued)(continued)

Scenario 5 (SoS Size Varies) Scenario 6 (SoS Size Varies)

Scenario 7-a (SoS Size = 10) Scenario 7-b (SoS Size = 100)



-15000.00

-10000.00

-5000.00

0.00

0 50 100 150 200 250

Number of Systems

Sav

ing

s (P

erso

n M

on

ths)

OSF 5%

OSF 10%

OSF 15%

Relative Cost of Collaborative and Acknowledged SoSECapability Affects All of the Systems


-10000.00

-8000.00

-6000.00

-4000.00

-2000.00

0.00

2000.00

0 50 100 150 200 250

Number of Systems

Sav

ing

s (P

erso

n M

on

ths)

OSF 5%

OSF 10%

OSF 15%

Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scope Varies


-1500.00

-1000.00

-500.00

0.00

500.00

1000.00

1500.00

0 1 2 3 4 5 6 7 8 9 10 11 12

Number of Systems Affected by Capability

Sa

vin

gs

(P

ers

on

Mo

nth

s)

OSF 5%

OSF 10%

OSF 15%



-5000.00

0.00

5000.00

10000.00

15000.00

20000.00

25000.00

0 20 40 60 80 100 120


Sav

ings

(Per

son

Mon

ths) OSF 5%

OSF 10%

OSF 15%



Scenario 8-a (SoS Size = 10) Scenario 8-b (SoS Size = 100)

Scenario 9 (SoS Size = 10) Scenario 10 (SoS Size = 5)


System Volatility = 1000 and SoS Capability = 1 Req

-300.00

-200.00

-100.00

0.00

100.00

0 1 2 3 4 5 6 7 8 9 10 11 12


Sav

ing

s (P

erso

n M

on

ths)

OSF 5%

OSF 10%

OSF 15%


System Volatility = None and SoS Capability = 1000 Reqs

-1000.00

-500.00

0.00

500.00

1000.00

1500.00

0 1 2 3 4 5 6 7 8 9 10 11 12


Sav

ing

s (P

erso

n M

on

ths)

OSF 5%

OSF 10%

OSF 15%

Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scopre Varies


0.00

5000.00

10000.00

15000.00

20000.00

25000.00

0 20 40 60 80 100 120

Number of SYstems Affected by Capability

Sav

ings

(Per

son

Mon

ths)

OSF 5%

OSF 10%

OSF 15%

Relative Cost of Collaborative and Acknowledged SoSESoS Size = 5 SoS Capability Scope Varies


-1000.00

-500.00

0.00

500.00

0 1 2 3 4 5 6


Sav

ing

s (P

erso

n

Mo

nth

s) OSF 5%

OSF 10%

OSF 15%



Scenario 11 (SoS Size = 5) Scenario 12 (SoS Size = 5)



-800.00

-600.00

-400.00

-200.00

0.00

200.00

0 1 2 3 4 5 6


Sav

ings

(Per

son

Mon

ths)

OSF 5%

OSF 10%

OSF 15%


System Volatility = 1000 and SoS Capability = 1 Req

-160.00

-120.00

-80.00

-40.00

0.00

0 1 2 3 4 5 6


Sa

vin

gs

(P

ers

on

Mo

nth

s)

OSF 5%

OSF 10%

OSF 15%


There exists a threshold where it is more cost effective to manage and engineer

changes to an SoS using an SoSE team and this threshold can be determined by modeling the SoS’ interdependency and

complexity characteristics.

When is it cost effective to create and empower an SoSE team to oversee and guide the

evolution of an SoS?

SoSE Model

Model parameters: SoS size Scope/size of SoS change CS volatility SoSE oversight

ConclusionsConclusions


Conclusions Conclusions (continued)(continued)

• SoSE team is cost effective when– SoS contains more than a “few” systems– SoS capability changes typically affect a “significant

percentage” of constituent systems– SoS capability requirements are a “significant percentage”

of the total requirements being addressed by constituent systems in an upgrade cycle

– SoS oversight activities and the rate of capability modifications/changes being implemented are sufficient to keep an SoSE team engaged (i.e., little-to-no slack time)

• SoSE team is NOT cost effective when– The number of systems in an SoS is “small”– The constituent system volatility is high and the SoS

changes are small


Conclusions Conclusions (continued)(continued)

• The “oversight factor” (the amount of effort spent by the SoSE team to monitor non-SoS changes in the constituent systems) is a key factor in determining the cost effectiveness of the SoSE team– More work is needed to determine a more accurate

“oversight factor”– This factor may be variable across multiple SoSs

• There may be reasons other than cost to engage an SoSE team– Importance of SoS– Critical SoS performance requirements requiring extensive

analysis at the SoS level


Future WorkFuture Work

• Expand SoSE model to– Include schedule factors to allow trade-offs between

“faster” and “cheaper”

– Include quality factors based on complexities and the resulting rework due to inadequate SoS engineering

– Allow users to specify specific constituent system configurations to allow capability alternative trade-offs

• Investigate the factors in going from an Acknowledged SoS to a Directed SoS


Backup ChartsBackup Charts


Translating capability objectives Translating capability objectives

Translating capability objectives

Addressing new requirements

& options

Addressing new requirements

& options

Addressingrequirements

& solution options

Understanding systems &

relationships(includes plans)


relationships(includes plans)


relationships

External Environment

Developing, evolving and maintaining

SoS design/arch

Developing, evolving and maintaining

SoS design/arch

Developing& evolving

SoS architecture

Assessing (actual)

performance to capability objectives

Assessing (actual)

performance to capability objectives

Assessing performance to capability objectives

Orchestrating upgrades

to SoS


to SoS


to SoS

Monitoring & assessing

changes


changes


changes

Traditional SE and SoSE ActivitiesTraditional SE and SoSE Activities

Traditional SE (Defense Acquisition Guide

[DoD, 2006] View)

SoSE (SoS SE Guidebook View Based on

Interviews and Analysis of 18 DoD SoSs in Various Stages)


Key COSOSIMO Research FindingsKey COSOSIMO Research Findings

• Limitations of COSYSMO for “Directed” SoSE effort estimation– Missing cost factors

• Cost/schedule compatibility of proposed SE approach• Level of overall risk resolution• Number of constituent systems and associated organizations• Constituent system maturity and stability• Constituent system readiness

– Need to adjust for SoSE oversight of constituent system SE– Need ability to assign different EMs to various parts of SE


Using System Dynamics Models to Explore Using System Dynamics Models to Explore Alternatives or Influences in the Development of Large Alternatives or Influences in the Development of Large

Software-Intensive SystemsSoftware-Intensive Systems

• System dynamics modeling tool: visual modeling tools that allow one to conceptualize, simulate and analyze models of dynamic systems and processes

• Consist of causal loops or stock and flow diagrams

• Models are executable, allowing use to explore behaviors of the model as variables representing process influences are changed

• Examples:– Hybrid/plan-driven ICM [Madachy

et al., 2007]– Intergovernmental collaboration

[Cresswell et al., 2002] – Inter-Organizational Baseline

Alignment [Greer et al., 2005]– Requirements volatility [Ferreira,

2002]– Under-allocation of resources in

early phases of a project [Black and Repenning, 2001]

– Interactions between concurrently developed projects [Ford and Sterman, 2003]


Model Validity Rationale Model Validity Rationale

• SoSE model description– Comparison model based on a modified version of the validated

academic systems engineering cost model, COSYSMO

– Modifications based upon key findings of the OSD SoSE case studies

• Validation goal: Show that the SoSE cost model is a valid method conducting sensitivity analyses for two different SoS management strategies– Collaborative

– Acknowledged

• Not part of the validation goal: The estimation of actual effort associated with a specific SoS or a given set of SoSs– The calibration/validation of the SoSE model for this purpose is left for

future work


Model Validity Rationale Model Validity Rationale (continued)(continued)

• Validity argument– COCOMO II and Academic COSYSMO are multiple regression models

that have been calibrated and validated with actual data from primarily DoD programs

– Academic COSYSMO calibration data contains 3 SoS data points

– Most other COSYSMO calibration data points interface to other systems, which implies that they are part of one or more SoSs

– SoSE model was developed using • Academic COSYSMO that includes ability to distribute effort across SE phases• Locally validated COSYSMO extension to adjust effort for reuse/oversight of

evolving system components• COCOMO II technique for using multiple effort multipliers to characterize

components with different characteristics and complexities

– SoSE model parameters • Based on ranges of size drivers determined through case studies and surveys• Uses nominal cost driver values unless reasons identified in SoSE or SE

survey data to indicate otherwise• Resulting in a relative comparison of the two management approaches


Model Validity Rationale Model Validity Rationale (continued)(continued)

• Validity argument (continued)– The prediction accuracies (PRED factors) are

• COCOMO [Clark and Reifer, 2007]– PRED(30) = 75% (with no stratification of projects)– PRED(30) = 80% (with stratification of projects)

• COSYSMO [Valerdi, 2005]– PRED(30) = 75% (with stratification of projects)– PRED(30) = 85% (anecdotal evidence from local calibrations)

– The OSD SoSE cases studies show that SoS systems engineers perform the same types of activities as addressed by the SE cost model, COSYSMO

– The OSD SoSE case studies identify differences between SoSE and SE for a single system and most of these differences are with respect to parameters in the SE cost model, COSYSMO

– There exists a local (single organization) calibrated and validated method within COSYSMO to estimate effort for oversight of related/interfacing systems or reusable components [Wang et al, 2008]


ReferencesReferences

Ackoff, R. (1971); “Towards a System of Systems Concepts”, Management Science, Vol 17, No. 11, Theory Series, pp. 661-671.ANSI/EIA (1999). ANSI/EIA-632-1988 Processes for Engineering a System.Berry, B. (1964); “Cities as Systems within Systems of Cities”, The Regional Science Association Papers, Volume 13..Black, L. and Repenning, N. (2001); “Why Firefighting is Never Enough: Preserving High-Quality Product Development”, System Dynamics

Review, Vol. 17, No: 1, pp. 33-62Blanchard, B. and Fabrycky, W. (1998). Systems Engineering and Analysis, Prentice Hall.Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. J. and Steece, B. (2000). Software Cost

Estimation With COCOMO II, Prentice Hall.Boehm, B. and Lane J. (2006) "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk - The

Journal of Defense Software Engineering, Vol. 19, No. 5, pp.4-9.Boehm, B., Valerdi, R., Lane, J., Brown, A., (2005) “COCOMO Suite Methodology and Evolution,” CrossTalk - The Journal of Defense

Software Engineering, Vol. 18, No. 4, pp. 20-25, April 2005.Cocks, D. (2006); “How Should We Use the Term “System of Systems” and Why Should We Care?”, Proceedings of the 16th Annual INCOSE

International Symposium.Cresswell, A. et al. (2002); "Modeling Intergovernmental Collaboration: A System Dynamics Approach", Proceedings of the 35th Annual

Hawaii International Conference on System Sciences.Dahmann, J. and Baldwin. K. (2008); “Understanding the Current State of US Defense Systems of Systems and the Implications for Systems

Engineering”, Montreal, Canada: Proceedings of the IEEE Systems Conference, 7-10 April.Department of Defense (DoD) (2006); Defense Acquisition Guidebook, Version 1.6, accessed at http://akss.dau.mil/dag/ on 2/2/2007.Department of Defense (DoD) (2008); Systems Engineering Guide for System of Systems, version 1.0.Dorner, D. (1996); The Logic of Failure, Metropolitan Books.Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation, Arizona State

University.Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects", Concurrent

Engineering Research and Application (CERA) Journal.Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York.Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System Development Programs",

Proceedings of IEEE Aerospace Conference.Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset House Publishing.


References References (continued)(continued)

INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03.isee Systems (2007), "iThink", http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx accessed on 2/10/2007.ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes.Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation, Arizona

State University.Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects", Concurrent

Engineering Research and Application (CERA) Journal.Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York.Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System Development

Programs", Proceedings of IEEE Aerospace Conference.Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset House

Publishing.INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03.isee Systems (2007), "iThink", http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx accessed on 2/10/2007.ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes.Kreitman, K.(1996), "From 'The Magic Gig' to Reliable Organizations: A New Paradigm for the Control of Complex Systems",

Symposium on Complex Systems Engineering, http://cs.calstatela.edu/wiki/index.php/ Symposium_on_Complex_Systems_Engineering, accessed on 1/11/2007.

Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33Lane, J. and Boehm, B. (2007); Modern Tools to Support DoD Software Intensive System of Systems Cost Estimation: A DACS

State-of-the-Art Report, Data and Analysis Center for Software, DACS Report Number 347336.Lane, J. and Valerdi, R., (2007); "Synthesizing System-of-Systems Concepts for Use in Cost Estimation", Systems Engineering, Vol.

10, No. 4. Lu, S. (2003); Engineering as Collaborative Negotiation: A New Paradigm for Collaborative Engineering,

http://wisdom.usc.edu/ecn/about_ECN_what_is_ECN.htm accessed on 2/14/2007.Madachy, R., B. Boehm, and J. Lane (2007); “Assessing Hybrid Incremental Processes for SISOS Development”, Software Process:

Improvement and Practice, Vol. 12, Issue 5, pp. 461-473.Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284)


References References (continued)(continued)

NAVSTAR Global Positioning System Joint Program Office, http://gps.losangeles.af.mil/ , accessed on 12/6/2006.

Northrop, L., et al. (2006); Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute.

Pinney, B. (2001); Projects, Management, and Protean Times: Engineering Enterprise in the United States, 1870-1960, PhD Dissertation, Massachusetts Institute of Technology.

Pressman, J. and Wildavsky, A. (1973); Implementation: How Great Expectations in Washington are Dashed in Oakland; Or, Why It’s Amazing that Federal Programs Work at All, This Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes, University of California Press.

Rechtin, E. (1991); Systems Architecting: Creating & Building Complex Systems, Prentice Hall.

SEI (2001), Capability Maturity Model Integration (CMMI), CMU/SEI-2002-TR-001.

Sheard, S. (2006), "Foundations of Complexity Theory for Systems Engineering of Systems of Systems", Proceedings of the IEEE Conference on System of Systems Engineering.

United States Air Force (USAF) Scientific Advisory Board (SAB) (2005); Report on System-of-Systems Engineering for Air Force Capability Development; Public Release SAB-TR-05-04

Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California.

Valerdi, R. and Wheaton, M. (2005); "ANSI/EIA 632 as a Standardized WBS for COSYSMO", AIAA-2005-7373, Proceedings of the AIAA 5th Aviation, Technology, Integration, and Operations Conference, Arlington, Virginia.

Wang, G., Valerdi, R., Ankrum, A., Millar, C., and Roedler, G. (2008), "COSYSMO Reuse Extension", Proceedings of the 18th Annual International Symposium of INCOSE, The Netherlands.

Documents

When Do You Need Systems of Systems Engineering: A Quantitative Analysis Jo Ann Lane 17 March 2009 University of Southern California Center for Systems