A Case Study in Predictive Program Management · A Case Study in Predictive Program Management John Welsh, Bipin Chadha, Zach Horiatis Lockheed Martin Advanced Technology Laboratories

A Case Study in Predictive ProgramA Case Study in Predictive ProgramManagementManagement

John Welsh, Bipin Chadha, Zach HoriatisJohn Welsh, Bipin Chadha, Zach Horiatis

Lockheed MartinLockheed Martin

Advanced Technology LaboratoriesAdvanced Technology Laboratories

Mission CriticalMission CriticalEnterprise SystemsEnterprise SystemsSymposiumSymposium

Oct. 2004Oct. 2004

MCES 2004 October 26 – 28, 2004 Bipin Chadha Page 2

ProblemProblem Major programs oftenMajor programs often

encounter major cost andencounter major cost andschedule problems despiteschedule problems despitedetailed and comprehensivedetailed and comprehensiveplansplans

““Despite their superiority, weapon systemsDespite their superiority, weapon systemsroutinely take much longer to field, costroutinely take much longer to field, costmore to buy, and require more support thanmore to buy, and require more support thaninvestment plans provide for.investment plans provide for.””

Defense Acquisitions: Assessments of MajorDefense Acquisitions: Assessments of MajorWeapon Programs, GAO Report, GAO-03-476,Weapon Programs, GAO Report, GAO-03-476,May 2003May 2003

Complex interrelationships cause control of program cost andComplex interrelationships cause control of program cost andschedule to be difficultschedule to be difficult Program managers need a small set of key levers (vs. thousands of tasks,Program managers need a small set of key levers (vs. thousands of tasks,

schedules, budgets, risks, action items, schedules, budgets, risks, action items, ……))

““U.S. Navy and Air Force plans to stiffen penalties for cost overrunsU.S. Navy and Air Force plans to stiffen penalties for cost overruns””Bloomberg Bloomberg –– 2/5/2004 2/5/2004


Recognize complex impacts of change at the speed ofRecognize complex impacts of change at the speed ofinformation rather than speed of information rather than speed of ““status reportstatus report”” delivery delivery

What If You Could What If You Could ……

Know that a software project would be delayed beforeKnow that a software project would be delayed beforeit was reportedit was reported

See management data in real-time and play See management data in real-time and play ““what-ifswhat-ifs””

Recognize cross IPT shortfalls and impactsRecognize cross IPT shortfalls and impactsimmediatelyimmediately

See third and fourth level effects of change rather thanSee third and fourth level effects of change rather thanthose immediately apparentthose immediately apparent

Do multi-variant cost sensitivity analysis on the flyDo multi-variant cost sensitivity analysis on the fly


Levels of UnderstandingLevels of Understanding Events (Reactive)Events (Reactive)

This event happened therefore we must takeThis event happened therefore we must takethis actionthis action

Q3 earnings results announced leading toQ3 earnings results announced leading todrop in stock pricedrop in stock price

Patterns of behavior (Responsive)Patterns of behavior (Responsive) Interpretation of current events in the contextInterpretation of current events in the context

of long-term historical changeof long-term historical change Understand a pattern of behaviorUnderstand a pattern of behavior Product lifecycles follow the Product lifecycles follow the ““SS”” curve curve

Systemic structure (Generative)Systemic structure (Generative) Addresses the question - what causes theAddresses the question - what causes the

pattern of behaviorpattern of behavior Attempt to address the underlying causes ofAttempt to address the underlying causes of

behaviorbehavior-- Senge, P. , The Leader’s New Work: Building Learning Organizations, SMR, Fall, 90

Sta

tist

ical

Met

ho

ds

Sys

tem

ic M

eth

od

s


ThesisThesis Prediction on large complex programs is feasiblePrediction on large complex programs is feasible

because because ……

Most program Most program ““surprisessurprises”” result from a small number result from a small numberof key elementsof key elements DependenciesDependencies

Systemic behavior patternsSystemic behavior patterns

These are normally not considered in planningThese are normally not considered in planning Hence root causes of program surprise go undetectedHence root causes of program surprise go undetected

Information for effective analysis is straightforward /Information for effective analysis is straightforward /easy to obtaineasy to obtain

Analysis methodology tolerates parameter variationAnalysis methodology tolerates parameter variationand imprecisionand imprecision


Comprehensive Program ManagementComprehensive Program Management

22ndnd Integrated Management Integrated ManagementFrameworkFramework Linked data elementsLinked data elements Configuration managementConfiguration management WorkflowWorkflow Access controlAccess control Provides views acrossProvides views across

disciplines, supports changedisciplines, supports changemanagement and traceabilitymanagement and traceability

11stst Generation Program Management Generation Program Management IPTIPT’’s s assessments: Predictive but subjectiveassessments: Predictive but subjective Provides drilldown capabilityProvides drilldown capability

33rdrd Generation Predictive Program Management Generation Predictive Program Management System dynamic model for programSystem dynamic model for program

managementmanagement Causal relationships are modeledCausal relationships are modeled Agent assessments: Predictive and objectiveAgent assessments: Predictive and objective Provides Provides ““look ahead capabilitylook ahead capability”” and and ““what ifwhat if””

leverslevers

TechSched Risk Cost ...

Communicate

Sense

Act

Decide IMPIMS

Production

WBS

SOWCost

GFX

LogisticsXBR

Management

IPT1SE&I

Risk

Knowledge

TPMs

Requirements

Joint Analysis

IPT2

IPT3IPT4 IPT5

IPT6

Program Management

LSI

RequirementsRequirementsIMS/IMPIMS/IMP

TPMTPM RiskRisk

WBSWBS SOWSOW

CostCost

EVMSEVMS


Predictive Program ManagementPredictive Program ManagementEnvironmentEnvironment

Intelligent Mobile Agents

Collaborative Program Management Environment

Management Tools (EVMS, Schedule, etc.)

Prediction Models

StatisticalModels

Design StructureMatrices

System DynamicsModels

InformationQuality

Coupling andDynamics

Internet Backbone


Communicate

Sense

Act

Decide IMP

IMS

Production

WBS

SOWCost

GFX

LogisticsXBR

Management

IPT1

SE&I

Risk

Knowledge

TPMs

Requirements

Joint Analysis

IPT2

IPT3

IPT4 IPT5

IPT6

Program Management

LSI

System DynamicsModels

ProgramProgramPlanPlan

Program InformationProgram Information

TechSched Risk Cost TechSched Risk Cost

TodayToday

PredictedPredicted

PastPastHistoryHistory








ManagementTeam

What-IfsWhat-IfsRecommenderRecommenderAgentsAgents

Data Gathering/Data Gathering/MonitoringMonitoringAgentsAgents

AlertingAlertingAgentsAgents

FeedbackFeedback

TaskTaskMonitoringMonitoringAgentsAgents

Project ControlDashboard

EVEVScheduleScheduleRisk Risk ……

Architecture ConceptArchitecture Concept


Coupling / Interdependencies

• Program Elements

— Activities

— Systems

— Organizations

• Dependency Types

— Information

— Physical

— Energy

— Resource

Key Coupling FactorsKey Coupling Factors

Program Element

Pro

gra

m E

lem

ent

Dependency Type

Design Structure Matrix

Common dependency patterns inherent across programs enablea small number of models to represent the dominant behaviors.


A B C DABCD

AB

CDx

x xx

x

Task 4Task 3Task 2Task 1

A D C BA

DC

Bx

xx

xxA

BCDTask 4

Task 1

Task 2Task 3

ModulesA affects D

DSM Dependency AnalysisDSM Dependency Analysis For each dependency type, matrices are analyzed to find task orFor each dependency type, matrices are analyzed to find task or

system groupings that are most important to watchsystem groupings that are most important to watch Supports automated analysisSupports automated analysis Easy to interpret, visualize, and analyzeEasy to interpret, visualize, and analyze

Rows and columns: Rows and columns: Program elementsProgram elements Matrix entries: Matrix entries: Coupling strengthsCoupling strengths Principle eigenvalues: Key work / design modes; determine the Principle eigenvalues: Key work / design modes; determine the ““raterate

and nature of convergenceand nature of convergence”” in the presence of change in the presence of change Principle eigenvector: Ranks tasks/subsystems by their Principle eigenvector: Ranks tasks/subsystems by their ““relativerelative

impactimpact”” on overall schedule / cost on overall schedule / cost

Dependency analysis highlights the subsets programs whichrequire closer attention / monitoring, or need to be decoupled


DSM AnalysisDSM Analysis Design mode: Group of tasks that are closely relatedDesign mode: Group of tasks that are closely related

and cause rework. Characterized by eigenvalues andand cause rework. Characterized by eigenvalues andcorresponding eigenvectorscorresponding eigenvectors

Eigenvalues and Eigenvectors determine the rate andEigenvalues and Eigenvectors determine the rate andnature of convergence of processnature of convergence of process

Magnitude of eigenvalues identifies the geometric rateMagnitude of eigenvalues identifies the geometric rateof convergenceof convergence Large positive eigenvalues have greater impactLarge positive eigenvalues have greater impact

Ignore complex eigenvalues with negative real componentIgnore complex eigenvalues with negative real component

Negative and complex eigenvalues describe dampedNegative and complex eigenvalues describe dampedoscillationsoscillations

Eigenvectors characterize relative contribution ofEigenvectors characterize relative contribution oftasks to reworktasks to rework


Selected Design Modes can beSelected Design Modes can beAnalyzed for Systemic Behavior PatternsAnalyzed for Systemic Behavior Patterns

Dynamic PatternsDynamic Patterns Balancing LoopBalancing Loop

Reinforcing LoopReinforcing Loop

Hidden ReworkHidden Rework

Fixes that BackfireFixes that Backfire

Limits to GrowthLimits to Growth

Shifting the BurdenShifting the Burden

Tragedy of CommonsTragedy of Commons

Accidental AdversariesAccidental Adversaries

Success to the SuccessfulSuccess to the Successful

EscalationEscalation

Drifting GoalsDrifting Goals

The Attractiveness PrincipleThe Attractiveness Principle

Growth and UnderinvestmentGrowth and Underinvestment(core patterns)(core patterns)

Interdependencies /Interdependencies /CouplingCoupling

Program Elements Activities

Systems

Organizations

Dependency TypesDependency Types Informational

Physical

Energy

Resource

Common systemic patterns inherent across programsenable a small number of models to represent

most of the dominant behaviors (80% solution).


SYSTEMComp1

Comp2Comp3

Comp1

Comp2 Comp3

Systemic BehaviorSystemic Behavior Behavior of a system that emerges due to dynamicBehavior of a system that emerges due to dynamic

interactions among componentsinteractions among components

System characteristicsSystem characteristics System structure drives system behavior (especially theSystem structure drives system behavior (especially the

feedback loops)feedback loops) System exhibits properties and behaviors System exhibits properties and behaviors

different from the partsdifferent from the parts Everything is connected to everythingEverything is connected to everything There are no independent variablesThere are no independent variables Many effects are indirect and delayedMany effects are indirect and delayed Many dependencies are non-linearMany dependencies are non-linear Behavior is dynamicBehavior is dynamic

Systemic Behavior Can be a Significant Source of Risk and Surprise


System Dynamics Modeling (SDM)System Dynamics Modeling (SDM) ObservationObservation

Problem arises over and over againProblem arises over and over again You apply the fix, there is some relief,You apply the fix, there is some relief,

but the problem surfaces againbut the problem surfaces again

What is going onWhat is going on You are applying a symptomaticYou are applying a symptomatic

solution that alleviates symptomssolution that alleviates symptomsimmediately but exacerbates theimmediately but exacerbates theproblem over timeproblem over time

The problem causes symptoms toThe problem causes symptoms toreappear after some delayreappear after some delay

Since the fundamental solution has aSince the fundamental solution has adelay built into it (before you seedelay built into it (before you seeresults) it forces you to take theresults) it forces you to take thesymptomatic solution in the shortsymptomatic solution in the shortterm, thus trapping you in the viciousterm, thus trapping you in the viciouscyclecycle

Source:The Fifth Discipline Fieldbook by Senge, et al

“Shifting the Burden”Pattern

ProblemSymptom

SymptomaticSolution

SideEffect

B

R

FundamentalSolution

B

ProblemSymptom

SymptomaticSolution

SideEffect

B

R

FundamentalSolution

B


Where Have These TechniquesWhere Have These TechniquesBeen applied Before?Been applied Before?

DSMDSM Developed at MITDeveloped at MIT Applied in AutomotiveApplied in Automotive

industryindustry Applied on some DoDApplied on some DoD

programsprograms MIT runs regular coursesMIT runs regular courses

on it for programon it for programmanagersmanagers

Significant body ofSignificant body ofliterature exists thatliterature exists thatsupports this techniquesupports this technique

SDMSDM Developed at MITDeveloped at MIT Applied in a wide range ofApplied in a wide range of

industries to analyzeindustries to analyzesystemic problemssystemic problems

Applied on some DoDApplied on some DoDprogramsprograms

Significant body ofSignificant body ofliterature exists thatliterature exists thatsupports this techniquesupports this technique

Several patterns haveSeveral patterns havebeen well documentedbeen well documented

Individually both techniques have been proven useful, buthaven’t been tied into a comprehensive solution


Intelligent Mobile Agents for PMIntelligent Mobile Agents for PMAgent Tasks

Information gatheringInformation gathering Data analysis resulting in R/Y/G labelingData analysis resulting in R/Y/G labeling Determine each Determine each WBSWBS’’s s associated R/Y/Gassociated R/Y/G

cost / schedule / risk / etc.cost / schedule / risk / etc. Trend analysisTrend analysis Historical data analysisHistorical data analysis Action Item monitoringAction Item monitoring Critical path monitoringCritical path monitoring Generate alertsGenerate alerts Earned value analysisEarned value analysis Predictive analysisPredictive analysis RecommendationsRecommendations

Leveraging LockheedLeveraging LockheedMartin agent technology toMartin agent technology tomanage a SDM; user seesmanage a SDM; user sees

•• InformationInformation•• Not agentsNot agents•• Not volumes of dataNot volumes of data


Case Study:Case Study:DSM Based Rework AnalysisDSM Based Rework Analysis

DSM analysis shows one dominant design mode (eigenvalue) and relativeDSM analysis shows one dominant design mode (eigenvalue) and relativeimpact of tasks (eigenvector) on the overall scheduleimpact of tasks (eigenvector) on the overall schedule

Analysis is robust in the presence of parameter variation / uncertaintyAnalysis is robust in the presence of parameter variation / uncertainty Analysis predicts approximate percent rework for the tasks due toAnalysis predicts approximate percent rework for the tasks due to

dependencies / couplingdependencies / coupling Program manager can use this analysis to verify estimated reworkProgram manager can use this analysis to verify estimated rework Analysis can be used to evaluate the impact of task restructuring on expected reworkAnalysis can be used to evaluate the impact of task restructuring on expected rework

(which is often the cause of surprise towards the end)(which is often the cause of surprise towards the end)


Dependency

DependencyWith Delay

DSM can identify asmall subset ofdependencies thatrequire detailedmodeling via SDM

Case Study: DSM used to CreateCase Study: DSM used to CreateSystem Dynamics Model (SDM)System Dynamics Model (SDM)


Case Study: Model ResultsCase Study: Model Results(Actual (Actual vsvs. Predicted). Predicted)

Model is successfully predicting the observed data.

Level 1 Problem Reports

Level 2 Problem Reports

Planned

Predicted

Actual

Linear Plan

Non-Linear Results

Crossover Point


ConclusionsConclusions Systemic model demonstrates the behavior in theSystemic model demonstrates the behavior in the

presence of rework and feedbackpresence of rework and feedback

The results of the model are broadly applicableThe results of the model are broadly applicable All iterative processes will demonstrate this behavior to aAll iterative processes will demonstrate this behavior to a

certain extent (rework pattern)certain extent (rework pattern)

Model is successfully predicting the crossover effectModel is successfully predicting the crossover effect This is where the status goes from green to red overnightThis is where the status goes from green to red overnight

Early analysis can predict this effect and program managersEarly analysis can predict this effect and program managerscan plan accordingly to prevent surprisecan plan accordingly to prevent surprise

Model can be run anytime with latest data to detect possibilityModel can be run anytime with latest data to detect possibilityof crossovers (ideal task for an agent)of crossovers (ideal task for an agent)

Planning to deploy capability on programsPlanning to deploy capability on programs

Documents

A Case Study in Predictive Program Management · A Case Study in Predictive Program Management John Welsh, Bipin Chadha, Zach Horiatis Lockheed Martin Advanced Technology Laboratories