New Processes and Estimation Methods for Acquiring 21st Century Software-Intensive Systems of Systems for CS 510 Barry Boehm [email protected]@cse.usc.edu

New Processes and Estimation Methods for Acquiring 21st Century Software-Intensive Systems of Systems for CS 510 Barry Boehm [email protected]@cse.usc.edu Jo Ann Lane [email protected]@usc.edu Winsor Brown [email protected]@cse.usc.edu COCOMO Forum October 2005 USC CSE 2005 University of Southern California Center for Software Engineering

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20052 Overview Characteristics of 21st century software-intensive systems of systems (SISOS) Major SISOS acquisition risks Addressal via risk-driven spiral model Associated SISOS estimation challenges New processes for 21st century SISOS New estimation methods for 21st century SISOS Case study

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20053 What is a System-of-Systems? Very-large systems developed by creating a framework or architecture to integrate Existing systems Systems currently under development New systems to be developed SoS system components independently developed and managed Some outsourced by LSI Some externally evolving Business Domain: enterprise-wide and corss- enterprise integration to support core business enterprise operations across functional and geographical areas Military Domain: dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment SoS activities often planned and coordinated by a Lead System Integrator (LSI)

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20054 What is a Lead System Integrator? Organization (or set of organizations) selected to accomplish the definition and acquisition of SoS components, and the continuing integration, test, and evolution of the components and SoS Typical activities Lead concurrent engineering of requirements, architecture, and plans Identify and evaluate technologies to be integrated Conduct source selection Coordinate supplier activities and validate SoS architecture feasibility Integrate and test SoS-level capabilities Manage changes at the SoS level and across the SoS-related IPTs Manage evolving interfaces to external systems Typically do not develop system components to be integrated (possible exception: SoS infrastructure)

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20055 What Does an SISOS Look Like: Future Combat Systems

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20056 What Does an SISOS Look Like: Network Centric Operations and Warfare

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20057 The Need for Software-Intensive Systems of Systems (SISOS) Lack of integration among stove-piped systems causes Unacceptable delays in service Uncoordinated and conflicting plans Ineffective or dangerous decisions Inability to cope with fast-moving events Increasing SISOS benefits See first; understand first; act first Network-centric operations coordination Transformation of business/mission potential Interoperability via Integrated Enterprise Architectures

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20058 Integrated Enterprise Architectures DOD Architectural Framework (DODAF) Federal Enterprise Architectural Framework (FEAF) Zachman Framework

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 20059 System Acquisition Trends Traditional Acquisition Standalone systems Stable requirements Requirements determine capabilities Control over evolution Enough time to keep stable Failures locally critical Reductionist systems Repeatability-oriented process, maturity models Current/Future Trends Everything connected (maybe) Rapid requirements change COTS capabilities determine requirements No control over COTS evolution Ever-decreasing cycle times Failures globally critical Complex, adaptive, emergent SoSs Adaptive process models Result: Sequential acquisition practices are increasingly inadequate Fixed-requirements, -cost, -schedule contracting Waterfall legacies (e.g., MIL-STD-1521B, Reviews and Audits)

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200510 Complexity of Solution Spaces Size: 10-100 MLOC Number of external interfaces: 30-300 Number of Coopetitive suppliers: 20-200 Even more separate work locations Depth of supplier hierarchy: 6-12 levels Number of coordination groups: 20-200 Reviews, changes, risks, requirements, architecture, standards, procedures, technologies, -ilities, integration, test, deployment, personnel, infrastructure, COTS, Key personnel spend 60 hours/week in meetings Unprecedentedness Emergence Rapid change

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200511 Complexity of Solution Spaces - Breadth, Depth, and Length Platform N Platform 1 Infra C4ISR Command and Control Situation Assessment Info Fusion Sensor Data Management Sensor Data Integration Sensors Sensor Components : 2008 2010 2012 2014 2016 1.0 2.0 3.0 4.0 5.0 Width Length Depth DOTMLPF Legend: DOTMLPF Doctrine, Organization, Training, Materiel, Leadership, Personnel, Facilities C4ISR Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200512 Need Simultaneous Agility and Discipline Discipline for planning and structure Foundations (architecture, organizations) Agility to handle the environment Rapid, continuous change Concurrency of development Many suppliers, coordination groups, external interfaces Use risk analysis to determine how much agility, discipline is enough

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200514 Top-10 Risks: Software-Intensive Systems of Systems - CrossTalk, May 2004 1.Acquisition management and staffing 2.Requirements/architecture feasibility 3.Achievable software schedules 4.Supplier integration 5.Adaptation to rapid change 6.Quality factor achievability and tradeoffs 7.Product integration and electronic upgrade 8.Software COTS and reuse feasibility 9.External interoperability 10.Technology readiness

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200515 -A COCOMO II Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward How Much Architecting Is Enough?

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200516 $100M $50M Arch. A: Custom many cache processors Arch. B: Modified Client-Server 12 3 4 5 Response Time (sec) Original Spec After Prototyping Available budget Effect of Unvalidated Requirements -15 Month Architecture Rework Delay

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200517 Effect of Unvalidated Software Schedules Original goal: 18,000 KSLOC in 7 years Initial COCOMO II, SEER runs showed infeasibility Estimated development schedule in months for closely coupled SW with size measured in equivalent KSLOC (thousands of source lines of code): Months =~ 5 * 3KSLOC - KSLOC3001000300010,000 - Months335072108 Solution approach: architect for decoupled parallel development; Schedule As Independent Variable (SAIV) process

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200518 1. Shared vision and expectations management 2. Feature prioritization 3. Schedule range estimation and core-capability determination - Top-priority features achievable within fixed schedule with 90% confidence 4. Architecting for ease of adding or dropping borderline-priority features - And for accommodating past-IOC directions of growth 5. Incremental development - Core capability as increment 1 6. Change and progress monitoring and control - Add or drop borderline-priority features to meet schedule Cross Talk, January 2002 (http://www.stsc.hill.af.mil/crosstalk) *Schedule As Independent Variable; Feature set as dependent variable. Also works for cost, schedule/cost/quality as independent variable. The SAIV* Process Model

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200519 Top-10 Risks: Software-Intensive Systems of Systems - CrossTalk, May 2004 1.Acquisition management and staffing 2.Requirements/architecture feasibility 3.Achievable software schedules 4.Supplier integration 5.Adaptation to rapid change 6.Quality factor achievability and tradeoffs 7.Product integration and electronic upgrade 8.Software COTS and reuse feasibility 9.External interoperability 10.Technology readiness

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200520 COTS Upgrade Synchronization and Obsolescence Many subcontractors means an increasing number of evolving COTS interfaces Aggressively-bid subcontracts can deliver obsolete COTS New COTS released every 8-9 months (GSAW) COTS unsupported after 3 releases (GSAW) An actual delivery: 120 COTS; 46% unsupported Emphasize COTS interoperability in source selection process Develop contract/subcontract provisions/incentives to ensure Consistency and interoperability across contractor and subcontractor- delivered COTS software products Such products are recent-release versions Develop a management tracking scheme for all COTS software products in all CSOS software elements Develop a strategy for synchronizing COTS upgrades

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200521 Spiral Development Definition A risk-driven process model generator Used to guide concurrent engineering Two distinguishing features: Cyclic approach for growing system definition Anchor point stakeholder- commitment milestones

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200522 Risk Patterns Determine Spiral Specifics Example 1: Boeing 777 System requirements, architecture well understood Major risks Assembly line, tooling rework; long-lead components; supplier interface mismatches Thorough PDR, CDR Many suppliers developing to inconsistent requirements, interfaces Complete, validate these early Some human interface, safety risks Early prototypes, analysis Risk-driven waterfall variant of spiral

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200523 Risk Patterns Determine Spiral Specifics Example 2: Business Decision Support System architecture provided by Enterprise Resource Planning (ERP) package Major risks Requirements not well understood; emergent with system use Human interface; decision- relevant data and processing Rapid requirements change Evolutionary development on ERP package

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200524 Risk Patterns Determine Spiral Specifics Example 3: Complex C4ISR Major risks Requirements not well understood; emergent with system use Architecture not well understood Many suppliers developing to inconsistent interfaces Ambitious schedule; rapid change Many technical, program, resource risks Full spiral development with anchor point milestones, increments Example: automated IFF Inception Elaboration Constr-1 Constr-2

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200525 Different Risk Patterns Generate Different Spiral Realizations SRR PDR Source Selection CDR Product Spec Completeness Product Completeness Full Spiral (M-S C4ISR) LCO LCA Iter. 0 Iter. 1 Iter. 2 Iter. 3 Evolutionary (ERP) Waterfall (777)

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200526 Need for Emergent (Evolutionary, Spiral) Processes Products and system requirements not pre- specifiable Selected COTS strengths and weaknesses Selected best-of-breed suppliers User IKIWISI (Ill know it when I see it) Emergent needs via usage Processes not pre-specifiable 1.Determine best COTS products 2.Compensate for COTS shortfalls

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200527 SISOS Challenges for Traditional Estimation Methods Overlapping increments Hard to classify effort, calibrate details Test I 1, develop I 2, design I 3 Cross-increment change management Massive concurrency Across breadth, depth, length, versions Multiple process types with different drivers Acquisition, systems engineering, COTS-based, agile, plan-driven Hard to determine dependencies Many potential sources of critical path slippage Rapid change Across breadth, depth, length, versions Estimating additional LSI effort, schedule 5%? 15%? 50%? Effort and schedule drivers

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200528 Schedule Estimation Very Challenging Synchronization of system schedules Accounting for slippages, slack No simple cube-root rules Only tasks on critical path count Overlaps, concurrency make critical path hard to determine Deterministic estimation is optimistic estimation Tradeoffs between hasty source selection and later integration delays

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200529 How much Architecting is Enough? - A COCOMO II Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200531 Future System Types and Processes Mainstream enterprise operation Domain-specific user programming on evolving ERP infrastructure Evolving product lines Software-intensive: business, public service, infrastructure Hardware-intensive: cars, buildings, devices, robots Scalable evolutionary spiral processes Evolving, complex, net-centric systems of systems Defense-, crisis-, transportation-management Scalable evolutionary spiral processes Unprecedented exploratory systems Bio-computing, nanotechnology, virtual reality Skill-intensive rapid prototyping

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200532 Emerging Scalable Spiral Process - For 21st century systems engineering and acquisition The C4ISR Metaphor for NCSOS Acquisition Role of OODA loops Example: Internet Spiral Example: FCS Win Win Spiral Model Risk-Driven Scalable Spiral Model Increment view Life-cycle view Role of anchor point milestones Acquisition management implications People management implications

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200533 Acquisition C4ISR Via Spiral OODA Loops Life Cycle Architecture Milestone for Cycle Decide on next-cycle capabilities, architecture upgrades, plans Stable specifications, COTS upgrades Development, integration, V&V, risk management plans Feasibility rationale Act on plans, specifications Keep development stabilized Change impact analysis, preparation for next cycle (mini- OODA loop) Orient with respect to stakeholders priorities, feasibility, risks Risk/Opportunity analysis Business case/mission analysis Prototypes, models, simulations Observe new/updated objectives, constraints, alternatives Usage monitoring Competition, technology, marketplace ISR Operate as current system Accept new system Example: ARPANet/Internet Spiral

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200534 The Internet Spiral Process Ref: USAF-SAB Information Architectures Study, 1994 Approved Internet Standard Approved Draft Standard Approved Proposed Standard Unapproved Proposed Standard IESG Approval IETF Review IETF Review IETF Review Working Group Review Full Implementation Widespread Implementation and Test Multiple Implementation and Test Equipment Working Group Evolution Unapproved Draft Standard Unapproved Internet Standard IESG = Internet Engineering Steering Group IETF = Internet Engineering Task Force

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200535 FCS WinWin Spiral Model 1b. Stakeholders Identify System Objectives, Constraints, & Priorities (OC&Ps); Alternative Solution Elements 1a. Identify Success-Critical Stakeholders 2a. Evaluate Alternatives with respect to OC&Ps 2b. Assess, Address Risks 3. Elaborate Product and Process Definition 4. Verify and Validate Product and Process Definitions Stakeholders Commitment 4 5 6 8 2 1 Stakeholders Review 7 3 L COL CA Build 2 Build 3 Progress Through Steps Bl 1 Driven By: Success- critical stakeholders win conditions Risk Management Spiral anchor point milestones Feasibility Rationale

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200536 Risk-Driven Scalable Spiral Model: Increment View

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200537 Risk-Driven Scalable Spiral Model: Increment View

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200538 Risk-Driven Scalable Spiral Model: Life Cycle View System Inception System Elaboration Agile DI 2 (OO&D) Rebaselining Plan-Driven DI 1 Construction (A) DI 1 V&V Plan-Driven DI 2 Construction (A) DI 2 V&V System LCASystem, DI 1 LCADI 2 B/L LCA DI 2 LCA Changes LCA: Life Cycle Architecture IOC: Initial Operational Capability OO&D: Observe, Orient and Decide V&V: Verification and Validation DI: Development Increment B/L: Baselined

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200539 Risk-Driven Scalable Spiral Model: Life Cycle View System Inception System Elaboration Agile DI 2 (OO&D) Rebaselining Plan-Driven DI 1 Construction (A) DI 1 V&V Agile DI 3 (OO&D) Rebaselining Plan-Driven DI 2 Construction (A) DI 2 V&V Agile DI 4 (OO&D) Rebaselining Plan-Driven DI 3 Construction (A) DI 3 V&V DI 1 Transn DI 1 Usage DI 2 Transn DI 2 Usage DI 3 Transn DI 3 Usage System LCA DI 3 LCA System, DI 1 LCADI 2 B/L LCADI 3 B/L LCADI 4 B/L LCA Update DI 2 LCA Changes DI 4 LCA... DI 1 IOC DI 3 IOC DI 2 IOC LCA: Life Cycle Architecture IOC: Initial Operational Capability OO&D: Observe, Orient and Decide V&V: Verification and Validation DI: Development Increment B/L: Baselined

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200540 LCO (MS A) and LCA (MS B) Anchor Points Pass/Fail Criteria A system built to the given architecture will Support the operational concept Satisfy the requirements Be faithful to the prototype(s) Be buildable within the budgets and schedules in the plan Show a viable business case Establish key stakeholders commitment to proceed LCO: True for at least one architecture LCA: True for the specific life cycle architecture; All major risks resolved or covered by a risk management plan

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200541 Spiral Feasibility Rationale Deliverable LCO, LCA reviews not just UML/PowerPoint charts Need to show evidence of product and process feasibility Evidence provided by prototypes, production code, benchmarks, models, simulations, analysis Sizing and cost/schedule model results for process feasibility Evidence provided in advance to LCO/LCA review team Key stakeholders, specialty experts Lack of evidence risks destabilizing the process Needs coverage by viable risk mitigation plan Key new progress metric Feasibility evidence progress vs. plans

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200542 Acquisition Management Implications - I 20th century build-to-spec contracting practices usable in part Good fit for stabilized-increments team But not for rebaselining, V&V teams Time & materials or equivalent Award fee based on cost/effectiveness These apply all the way down the supplier chain Need top-level award fee for cost-effective team balancing No stable distribution of effort

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200543 Acquisition Management Implications - II Dont skimp on system definition phases But avoid analysis-paralysis Use Feasibility evidence generation as progress metric Use more evidence-based source-selection processes Competitive exercise as proof of capability Preceded by multistage downselect Use Schedule/Cost as Independent Variable processes Prioritized features as dependent variable Top priority: transformational empowerment of acquisition corps Education, mentoring, tools, techniques

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200544 Staffing Implications Critical success factors for system development organizations Understanding, dealing with sources of system value And associated stakeholders, applications domains Software/systems architecting and engineering Creating and/or integrating system components Using risk to balance disciplined and agile processes Ability to continuously learn and adapt Getting the right people on the right teamsit takes different types of people to make a successful team Plan-driven teams: Thrive on order Agile teams: Thrive on chaos Verification and validation teams: Thrive on oversight All supported by a collaborative environment to encourage continuous learning, optimization, and adaptation

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200545 Overview Characteristics of 21st century software-intensive systems of systems (SISOS) Major SISOS acquisition risks Addressal via risk-driven spiral model Associated SISOS estimation challenges New processes for 21st century SISOS New estimation methods for 21st century SISOS Evolution of estimation methods Cost estimation Schedule estimation Case study

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200546 Evolution of Estimation Methods to Support SISOS Acquisition Activities Estimation approaches need to more closely reflect development approaches Processes, both agile and traditional Architecture Factors for Security Reliability Building for reuse COTS incompatibilities People factors Politics Timeliness of key decisions Vendor compatibility Experience levels Resource availability Need to integrate existing stove-pipe cost models to better capture relationships between various teams and activities Integrate SE, COTS, software development, hardware development/manufacturing Need to add in budgets for activities not covered in current cost models LSI Post development life cycle activities (e.g., installation/deployment, operation, maintenance) Estimates evolve over time, starting with upfront knowns, then expanding to include tasks identified as part of architecture decisions

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200547 Observations on How Processes Are Adapting to the SISOS Environment Traditional planning and scheduling Plan activities as independent projects Requires that up-front SISOS architecting be performed in sufficient detail to allow sub-projects to be somewhat independent of each other Requires that risk-driven processes be used to identify and manage risks early at SISOS and sub-project levels Blend traditional processes with more agile processes Plan for stabilized evolutionary increments Concurrently have agile change/risk/opportunity team Performs acquisition intelligence/surveillance/reconnaissance functions Rebaselines future increment solutions Plan for early and continual verification and validation (V&V) Competing priorities: use stakeholders to negotiate priorities with other on-going system component enhancements and maintenance

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200548 Observations on How Processes Are Adapting to the SISOS Environment (continued) Project monitoring and control Minimize impacts on key personnel Prioritize oversight areas Integrated project management Identify key cross-cutting processes for standardization Allow flexibility in other areas Let organizations to use their own proven processes Supplier organizations have been selected by the independent system component owner for their technical expertise and ability to produce Decision making process Need to reduce to the extent possible Number of required SISOS-level decisions Number of clearances or approvals required for each decision Studies indicate that the probability of success decreases as the number of required decision approvals increases

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200549 Observations on How Processes Are Adapting to the SISOS Environment (continued) Risk management Cross-cutting risks need to be managed and balanced across system and organizational boundaries Each risk needs a responsible owner and committed suppliers Risk portfolios and owners to manage cross-cutting risks Integrated product teams typically play a much larger role and have more responsibilities The people processes are at least as important as the technical processes Personal, organizational, and political motivations and priorities can impact the success of the project

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200550 SISOS Schedule Estimation Customer, Users LSI Agile LSI IPTs Agile Suppliers Agile Suppliers PD V&V LSI Integrators RFP, SOW, Evaluations, Contracting Effort/Staff Proposals Similar, with added change traffic from users Assess compatibility, short-falls Rework LCO LCA Packages at all levels COSOSIMO -like Assess sources of change; Negotiate rebaselined LCA 2 package at all levels COSOSIMO -like Similar, with added re- baselineing risks and rework Inception Elaboration Source SoS Selection Architecting Increment 1 Increments 2, n Develop to spec, V&V CORADMO -like Degree of Completeness risks, rework Proposal Feasibility LCOLCA LCA 1 IOC 1 Effort/staff at all levels risks, rework Risk-manage slow- performer, completeness risks, rework Integrate COSOSIMO -like LCA 2 shortfalls risks, rework Effort COSYSMO-like. Schedule = Effort/Staff Try to model ideal staff size LCA 2

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200551 SISOS Cost Estimation Many models exist to support estimation of various aspects of system development Systems engineering Software development COTS integration Hardware development and manufacturing Others are currently under development Lead system integrator effort Security

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200552 Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle System Engineering Cost Model: COSYSMO Addresses first four phases of the system engineering lifecycle (per ISO/IEC 15288) Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632) Developed with USC-CSE Corporate Affiliate sponsorship and INCOSE participation

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200553 How is Systems Engineering Defined? EIA/ANSI 632 Processes for Engineering a System: Acquisition and Supply Supply Process Acquisition Process Technical Management Planning Process Assessment Process Control Process System Design Requirements Definition Process Solution Definition Process Product Realization Implementation Process Transition to Use Process Technical Evaluation Systems Analysis Process Requirements Validation Process System Verification Process End Products Validation Process

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200554 COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms + 3 adjustment factors - Application factors -8 factors - Team factors -6 factors COSYSMO Operational Concept

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200555 MyCOSYSMO* Capabilities Jointly developed by USC/CSE and Raytheon Provides costing using local rates as well as effort Supports multiple levels of estimate formality/complexity Budgetary estimate Rough order of magnitude (ROM) Proposal Embeds local systems engineering program performance data Systems Engineering size and productivity Environmental data Local site salary grade profiles Provides for more consistent inputs and outputs, reduces variability * Developed by USC CSE Raytheon Affiliate, Gary Thomas

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200556 MyCOSYSMO* Capabilities (continued) Focus on risk, uncertainty Provides user friendly Interface and documentation Provides for local site expansion Size drivers and effort multipliers Site unique parameters Historical data collection mode * Developed by USC CSE Raytheon Affiliate, Gary Thomas

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200557 How Much Effort to Architect and Integrate a System of Systems? Systems developed by system contractors Total effort 3000 person-years System of systems integration functions SoS abstraction, architecting, source selection, systems acquisition, integration, test, change management effort How much to budget for integration? What factors make budget higher or lower? How to develop and validate an estimation model? System of Systems ? person-years (PY) Sensing 500 PY Vehicles 500 PY Common 400 PY Infrastructure 600 PY Command & Control 1000 PY

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200558 Constructive System-of-System Integration Cost Model (COSOSIMO) Parametric model to estimate the effort associated with the definition and integration of software- intensive system of systems components Includes at least one size driver and 6 exponential scale factors related to effort Targets input parameters that can be determined in early phases Goal is to have zero overlap with COCOMO II and COSYSMO

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200559 Scope of Proposed SoS Cost Model Characteristics of SoSs supported by cost model Strategically-oriented stakeholders interested in tradeoffs and costs Long-range architectural vision for SoS Developed and integrated by an LSI System component independence Size drivers and scale factors Based on product characteristics, processes that impact LSI effort, and LSI personnel experience and capabilities Size Drivers Scale Factors SoS Definition and Integration Effort Calibration COSOSIMO

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200560 SISOS Schedule Estimation: A Composite Approach Customer, Users LSI Agile LSI IPTs Agile Suppliers Agile Suppliers PD V&V LSI Integrators RFP, SOW, Evaluations, Contracting Effort/Staff Proposals Similar, with added change traffic from users Assess compatibility, short-falls Rework LCO LCA Packages at all levels COSOSIMO -like Assess sources of change; Negotiate rebaselined LCA 2 package at all levels COSOSIMO -like Similar, with added re- baselineing risks and rework Inception Elaboration Source SoS Selection Architecting Increment 1 Increments 2, n Develop to spec, V&V CORADMO -like Degree of Completeness risks, rework Proposal Feasibility LCOLCA LCA 1 IOC 1 Effort/staff at all levels risks, rework Risk-manage slow- performer, completeness risks, rework Integrate COSOSIMO -like LCA 2 shortfalls risks, rework Effort COSYSMO-like. Schedule = Effort/Staff Try to model ideal staff size LCA 2

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200561 Composite Approach: Inception Phase COSYSMO estimate of system size, effort Medium difficulty for simplicity Requirements: 5,000 * 1.0 = 5,000 Interfaces: 500 * 4.3 = 2,150 Algorithms: 50 * 6.5 = 325 Scenarios: 200*22.8 = 4,560 Total Size: 11,035 Plus 10% Rqts. Volatility 12,138 Nominal effort = 0.254 * (12138)**1.06 = 5420 PM

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200562 Composite Approach: Inception Phase -2 Nominal effort = 5420 person months (PM) Effort multipliers Low requirements understanding = 1.37 Low architecture understanding = 1.28 High technology risk = 1.32 Product * Nominal PM = 2.315 * 5420 = 12,546 PM COSYSMO Conceptual Phase Effort = 23% 7% Inception; 16% Elaboration Inception Phase Effort = 12,546 *.07 = 878 PM Inception Phase Schedule Months = 1.5 * cube root (878) = 14.4 months Average staff size = 878/14.4 = 61 people

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200563 Composite Approach: Elaboration Phase Sum of schedules for systems engineering, source selection, and post-selection rebaselining COSYSMO Elaboration effort = 12,546 *.16 = 2007 PM Systems engr. schedule = 1.5 * cube root (2007) = 19 months Average staff size = 2007/19 = 106 people Source selection schedule Preparation in parallel: no added schedule RFP finalization and publication: 1 month Proposal responses: 3 months Including prototypes, architecture, Feasibility Rationale Parallel evaluation, finalist selection: 2 months Finalist compatibility/feasibility, Q&A: 3 months Contracting: 1 month Total schedule: 10 months Teambuilding, LCA rebaselining: 6 months Total Elaboration Phase schedule: 35 months

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200564 Composite Approach: Construction Phase Multiple development increments Usually 18-24 months for broad, deep SISOS Increments sequenced by user needs, cross-dependencies Assume 18 months plus LSI integration time COSOSIMO effort; cube root schedule estimator Scope supplier increments to allow some slack for synchronization and stabilization And interaction with V&V, agile teams Agile team schedule fixed Use COSYSMO Construction effort to baseline team size OT&E, Transition schedules Can do all but final increment in parallel with next increment Use experience on earlier increments to schedule final increment

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200566 Case Study Inception, Elaboration phase effort and schedule estimates for metropolitan area crisis management system Using overall approach presented above See handout for case study overview and analysis

Backup Charts

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200568 Acquisition Management and Staffing: Effect of Software Under-Representation Software risks discovered too late Slow, buggy change management Recent large project reorganization SW Software SW

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200569 Need for CRACK Integrated Team Members - CrossTalk, December 2003 Not Collaborative: Discord, frustration, loss of morale Not Representative: Delivery of unacceptable systems, late rework Not Authorized: Authorization delays, unsupported systems Not Committed: Missed action items, discontinuities, delays Not Knowledgeable: Unacceptable systems, delays, late rework

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200570 Supplier Integration: Rapid Adaptability to Change Depth of supplier chain increases communication and coordination effort Inflexible subcontracting is a major source of delays and shortfalls Develop subcontract provisions enabling flexibility in evolving deliverables. Develop an award fee structure based on objective criteria for: Schedule Preservation Cost Containment Technical Performance Architecture and COTS Compatibility Continuous Integration Support Program Management Risk Management

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200571 Rapid Adaptability to Change: Architecture Architecture may be over-optimized for performance vs. adaptability to change Modularize architecture around foreseeable sources of change Identify foreseeable sources of change Technology, interfaces, pre-planned product improvements Encapsulate sources of change within software modules Change effects confined to single module Not a total silver bullet, but incrementally much better

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200572 Rapid Adaptability to Change: Evolving Software Architecture Software architecture will need to change and adapt to rapidly changing priorities and architecture drivers New COTS releases Evolving enterprise standards and policies Emerging technologies and competitor threats Organize software effort to ensure the ability to rapidly analyze, develop, and implement software architecture changes Empower a focal-point integrator of the software architecture and owner of the critical software infrastructure Raise to a very high organizational level the Owner of the software architecture and infrastructure Owner of SOS software integration and test

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200573 Quality Factor Achievability and Tradeoffs Quality factor (-ility) tradeoffs are success-critical, difficult to analyze, and incompletely formulated These tradeoffs address competing requirements for performance, interoperability, security, safety, survivability, usability, modifiability, portability, reusability, accuracy and other attributes Create critical-mass subcontracts for software tradeoff analysis organizations to conduct continuing modeling, simulation, and execution analyses of success-critical quality factor tradeoff issues

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200574 Rapid, Synchronous Software Upgrades Out-of-synchronization software upgrades can be a major source of operational losses Software crashes, communication node outages, out-of-synch data, mistaken decisions Extremely difficult to synchronize multi-version, distributed, mobile-platform software upgrades Especially if continuous-operation upgrades needed Architect software to accommodate continuous-operation, synchronous upgrades E.g., parallel operation of old and new releases while validating synchronous upgrade Develop operational procedures for synchronous upgrades in software support plans Validate synchronous upgrade achievement in operational test and evaluation

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200575 COTS: The Future is Here Escalate COTS priorities for research, staffing, education Its not all about programming anymore New processes required

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200576 SISOS Compound Software Risks Serious inter-system compound risks will be discovered late Compound risks are frequently architecture-breakers, budget- breakers, and schedule-breakers Examples include closely-coupled immature technologies and closely-coupled, ambitious critical path tasks Establish a hierarchical software risk tracking compound risk assessment scheme The top level is Top-10 system-wide software risks Tier down to system-level and subcontractor-level Top-10 risk lists Valuable both for overall software risk management and for compound risk assessment.

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200577 SISOS Compound Software Risks (continued) Develop high-priority plans to decouple high-risk elements and to reduce their risk exposure Establish a SISOS Software Risk Experience Base This is extremely valuable in avoiding future instances of previously experienced risks

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200578 Failure Mode I: Build-to-Spec DeliverablesPurchasing agent metaphor Rapid change: heavy spec change traffic, slow contract changes Plus deep supplier chain: slowdowns multiply, changes interact Plus emergent requirements: many initial specs wrong; more changes Plus build-to-spec award fee: supplier inertia Bottom line: late rework, overruns, mission shortfalls

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200579 Why Software Projects Fail Overruns: 189% cost; 222% schedule; 61% of product

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200580 18-24 Month Development Increments Need More Concurrency

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200581 Failure Mode II: Sequential Document-Driven MilestonesWaterfall, V-model, MIL-STD-1521B Requirements emergence, COTS: freeze requirements too early Plus document-completion progress metrics: hasty point solutions, undiscovered risks Plus rapid change: problems with Failure Mode I Bottom line: more late rework, overruns, mission shortfalls

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200582 $100M $50M Required Architecture: Custom; many cache processors Original Architecture: Modified Client-Server 12 3 4 5 Response Time (sec) Original Spec After Prototyping Original Cost The Cost of Hasty Specifications 15-Month Architecture Rework Delay

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200583 Failure Mode III: Hasty Best-of-Breed Source Selection Complexity, emergence: incomplete, unvalidated system architecture, solicitation SOWs Deep, wide supplier chains: incompatible legacy solutions, COTS; critical-path modeling and simulation needs Rapid change: rapid COTS evaluation, version obsolescence GSAW: 8-11 months/release; 3 supported releases Bottom line: serious integration problems, overruns, mission shortfalls Over-ambitious startup schedule

Acquiring 21st Century SISOS USC CSE 2005 COCOMO Forum 200584 Failure Mode IV: Incremental Document-Driven Development High assurance of ilities: deferral to later increments Complex NCSOS: early-increment architecture inadequate for later-increment ilities. Rapid change: destabilization of ambitious increment schedules; increment completion delays; next increment destabilized Bottom line: serious security, safety, scalability problems; destabilized development; more rework, overruns, shortfalls Risk-insensitive spirals; ambitious increment schedules

Documents

New Processes and Estimation Methods for Acquiring 21st Century Software-Intensive Systems of Systems for CS 510 Barry Boehm [email protected]@cse.usc.edu