How to Assess a Joint Confidence Level (JCL) Model

Brian Berry

2017 NASA Cost and Schedule Symposium

August 29 – 31

Washington, D.C.

Outline

Background

JCL Process

NASA JCL Modeling Lessons Learned

JCL Model Assessment

Identify and track model and input changes over time

Validate calculations

Evaluate Active Risks

Evaluate deterministic critical path and probabilistic critical path and tasks

Quickly access/understand model results

Summary

2017 NASA Cost and Schedule Symposium 2

Background

Numerous JCL analyses have been performed since NASA policy established JCL requirement in 2009

Many lessons learned with improvements to processes, tools, and analysis

70% JCL value is a high bar metric—difficult to meet

JCL model output value can become focus of analysis

Analysts need eyes wide open during entire process

Need to assess model, inputs, and outputs to help ensure the JCL value is robust

2017 NASA Cost and Schedule Symposium 3

JCL Process*

0. Identify goals of the JCL analysis

1. Develop a summary analysis schedule

2. Load cost onto the schedule activities

3. Incorporate the risk list

4. Conduct an uncertainty analysis

5. Calculate and view the results

2017 NASA Cost and Schedule Symposium

* NASA Cost Estimating Handbook V4.0, pJ-11

4

NASA JCL Modeling Lessons Learned

Durations that may be lengthy (> 44 days per best practices) can be acceptable for JCL analysis schedules

Analysis schedules must be structurally sound with solid logic and be accurately statused. For use in SRAs and JCLs the network must be able to move freely without limitations of constraints

All main phases of the schedule are linked together so as durations fluctuate during simulation, the schedule on a whole expands and contracts appropriately and no unnecessary constraintsLinkage (relationship) types are important—Strive for >90% FS and avoid tempting shortcuts such as creating many Start-To-Start linkages with lags to mimic Finish-To-Start

JACS Reporting capabilities can assist in model verification Preds tool-see what tasks each task feeds—should support milestone

Cost contributors chart-delta between estimate & mean should reflect intended cost uncertaintyCriticality Index-ensure relative differences between tasks makes senseRisk Event Ranking-Use Prob./Impact outputs to understand model impacts

2017 NASA Cost and Schedule Symposium 5

NASA Emphasized JACS Development and Reports Features to Support Key Elements of the JCL Analysis Lifecycle

Assess Model Quality

2017 NASA Cost and Schedule Symposium

Health ChecksJCL Inputs

Identify Model Changes Model Comparison

Validate Calculations Baseline Comparison

Insight into Model Results Critical Path ReportInsight Application

Compare Scenario Results Insight Report Compare

6

Assessing Model Quality

Key to a JCL Model is the underlying Schedule Structure and Cost, Risk, and Uncertainty Inputs

Several features were implemented in JACS to provide rapid access to these areas

Health Checks

– Checks quality of underlying schedule and reviews syntax, logic, and specification of JCL model inputs to ensure they are proper

JCL Inputs

– Provides insight into application of JCL uncertainty distributions and risks to the underlying schedule and cost

2017 NASA Cost and Schedule Symposium 7

Health Checks

Health Check Report Includes GAO Best

PracticesDCMA 14-Pt AssessmentJACS HealthIssue CountsIssue List

Provides many ways to help ensure that a sound schedule, uncertainty, and cost loading will underpin the JCL

2017 NASA Cost and Schedule Symposium 8

Task

ID

Unique

Task ID Name Task Details Uncertainty Correlation Group Task Type Status Start Finish

Duration

(Days) % Complete

Remaining

Duration Constraint

Constraint

Date

Actual

Start

Actual

Finish Predecessors Successors

WBS

Number

Total Cost

($)

TI Cost

($)

TD Cost

($)

4 331 Delayed Award Activity With Duration UncertaintyN/A Activity Future Task 10/1/2013 12/2/2013 45 0.00% 45 ASAP 10/1/2013 NA NA 3 5 1.1.2 $0 $0 $0

12 16 EMD HW System Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 12/3/2013 8/4/2014 175 0.00% 175 ASAP 10/1/2013 NA NA 5 7,27,52 1.4.1.1 $2,400 $720 $1,680

13 15 EMD HW Initial Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 8/5/2014 4/13/2015 180 0.00% 180 ASAP 10/1/2013 NA NA 7 8,30,52 1.4.1.2 $4,800 $1,440 $3,360

14 14 EMD HW Detailed Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 4/14/2015 5/16/2016 285 0.00% 285 ASAP 10/1/2013 NA NA 8 9,51,52 1.4.1.3 $9,000 $2,700 $6,300

15 13 EMD HW Final Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 5/17/2016 8/7/2017 320 0.00% 320 ASAP 10/1/2013 NA NA 9 32,25FF,16,51,521.4.1.4 $5,200 $1,560 $3,640

18 78 EMD SW System Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 12/3/2013 8/4/2014 175 0.00% 175 ASAP 10/1/2013 NA NA 5 7,28,53 1.4.2.1 $6,150 $1,538 $4,613

19 83 EMD SW Initial Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 8/5/2014 4/13/2015 180 0.00% 180 ASAP 10/1/2013 NA NA 7 8,31,53 1.4.2.2 $12,300 $4,920 $7,380

20 88 EMD SW Detailed Design Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 4/14/2015 5/16/2016 285 0.00% 285 ASAP 10/1/2013 NA NA 8 9,21,51,53 1.4.2.3 $21,600 $5,400 $16,200

22 110 EMD SW Refinements Activity With Duration UncertaintyEMDDURATION, EMDCOSTSActivity Future Task 5/17/2016 1/15/2018 435 0.00% 435 ASAP 10/1/2013 NA NA 9 32,25FF,51,53 1.4.2.5 $19,450 $3,890 $15,560

36 313 Prod Procure Materials Activity With Duration UncertaintyPRODDURATION, PRODCOSTSActivity Future Task 1/16/2018 1/27/2020 530 0.00% 530 ASAP 10/1/2013 NA NA 34 37SS+140Days,49,541.5.2.1 $28,700 $20,090 $8,610

37 314 Prod Air Vehicle Activity With Duration UncertaintyPRODDURATION, PRODCOSTSActivity Future Task 7/31/2018 6/14/2021 750 0.00% 750 ASAP 10/1/2013 NA NA 36SS+140Days 38SS+80Days,49,48FF,541.5.2.2 $48,650 $43,785 $4,865

38 316 Prod IAT&C Activity With Duration UncertaintyPRODDURATION, PRODCOSTSActivity Future Task 11/20/2018 3/8/2021 600 0.00% 600 ASAP 10/1/2013 NA NA 37SS+80Days 41SS,42SS,40SS+180Days,39,48FF,45FF+45Days1.5.2.3 $16,426 $0 $16,426

40 317 Prod ST&E Activity With Duration UncertaintyPRODDURATION, PRODCOSTSActivity Future Task 7/30/2019 7/26/2021 520 0.00% 520 ASAP 10/1/2013 NA NA 38SS+180Days 48FF,49 1.5.3 $8,000 $6,400 $1,600

41 319 Prod Common Support EquipmentActivity With Duration UncertaintyPRODDURATION, PRODCOSTSActivity Future Task 11/20/2018 12/16/2019 280 0.00% 280 ASAP 10/1/2013 NA NA 38SS 49,54 1.5.4 $12,000 $9,600 $2,400

42 320 Prod Peculiar Support EquipmentActivity With Duration UncertaintyPRODDURATION, PRODCOSTSActivity Future Task 11/20/2018 10/21/2019 240 0.00% 240 ASAP 10/1/2013 NA NA 38SS 49,54 1.5.5 $16,500 $14,025 $2,475

Inputs Details

Schedule Statistics

Description Count % of Total Count % of Total

Total Lines in IMS 26 48% 28 52%

Total Activities 15 100% 0 0%

Total Milestones 11 44% 14 56%

Schedule Milestones 0 0% 8 100%

Risk Events 7 100% 0 0%

Risk Factors 4 100% 0 0%

Start/Finish Hammock Milestones 0 0% 6 100%

Total # of Summary tasks 0 0% 14 100%

Schedule Summary Tasks 0 0% 8 100%

Hammock tasks 0 0% 3 100%

Multi-Impact Risk Events 0 0% 2 100%

Risk Factor Summary Section Heading 0 0% 1 100%

COMPLETED Activities 0 0% 0 0%

IN PROGRESS Detail Activities 0 0% 0 0%

Activities with <5 workdays remaining 0 0% 0 0%

Activities with 6 to 20 workdays remaining 0 0% 0 0%

Activities with 20 to 100 workdays remaining 0 0% 0 0%

Task with >100 workdays remaining 0 0% 0 0%

FUTURE Detail Activities 15 100% 0 0%

Activities with <5 workdays duration 0 0% 0 0%

Activities with 6 to 20 workdays duration 0 0% 0 0%

Activities with 20 to 100 workdays duration 1 100% 0 0%

Task with >100 workdays duration 14 100% 0 0%

COST STATISTICS Count % of Total Count % of Total

Total Lines in IMS 27 50% 27 50%

Total Activities 14 93% 1 7%

Total Milestones 10 40% 15 60%

Total # of Summary tasks 3 21% 11 79%

Total # of Hammock tasks 3 100% 0 0%

Total number of Complete Cost loaded Tasks 0 0% 0 0%

Total number of In Progress and Future Cost loaded Tasks 14 100% 0 0%

Tasks with only Time-Independent (TI) Cost Specified 0 0% 0 0%

Tasks with only Time-Dependent (TD) Cost Specified 1 100% 0 0%

Tasks with both TI and TD Cost Specified 13 100% 0 0%

Point Estimate Cost

Sum of all TI cost

Sum of all TD cost

Uncertainty OverviewBOTH Duration and Cost

RISK EVENT/FACTOR STATISTICS Count % of Total Count % of Total Count % of Total Count % of Total

Total Number of Risk Events/Factors 13 100% 10 77% 1 8% 0 0%

Number of RISK EVENTS 9 69% 7 78% 0 0% 0 0%

Likelihood of occurrence <25% 4 44% 2 50% 0 0% 0 0%

Likelihood of occurrence 25% to 75% 1 11% 1 100% 0 0% 0 0%

Likelihood of occurrence > 75% 4 44% 4 100% 0 0% 0 0%

Number of RISK FACTORS 4 31% 3 75% 1 25% 0 0%

Likelihood of occurrence <25% 4 100% 3 75% 1 25% 0 0%

Likelihood of occurrence 25% to 75% 0 0% 0 0% 0 0% 0 0%

Likelihood of occurrence > 75% 0 0% 0 0% 0 0% 0 0%

With COST Uncertainty OnlyTotal

WITHOUT Cost Uncertainty

Count % of Total

0%

14 26%

3 6%

Total

54 100%

15 28%

0

15 28%

25 46%

0

0%

7%

16%

24%

7

4

WITHOUT Duration UncertaintyTotal

Count % of Total

87%

0

7%1

14

25 46%

1

14

93%

100%

14

WITH Duration Uncertainty

13

54 100%

0

0

1

8

With DURATION Uncertainty Only

0%0

21%

0%

0%

0%

7%

93%

0

0

0%

0%

0%

0

$267,476.000 100%

$116,067.500 43%

$151,408.500 57%

WITH Cost Uncertainty

26%

2

8 57%

14%

3

15

6

32%

28%

0%

0

Inputs

Summary

JCL Inputs

Comprehensive metrics/information on

Baseline Schedule

Risk events, risk factors

Cost Loading

Uncertainty

Correlation

Goes beyond basic health characteristics to assess reasonableness of inputs

2017 NASA Cost and Schedule Symposium 9

Identifying Model Changes

JCL models go through several changes during their lifecycle During project development the underlying schedule structure may change

During drops to the SRB leading up to the final review the schedule and modeling inputs may changeAfter model is provided to SRB, SRB analysts may change model inputs (e.g., uncertainty distributions, risk attributes, add/delete risks)

The JACS development team assessed the requirement and identified the need to compare differences in schedule structure and model inputs Design was based on adding functionality complementary (not supplementary) to

MS Project– MS Project desktop does not have a model tracking feature

MS Project Server has functionality to conduct journal tracking if the feature is enabled and server database is structured appropriately

–

Design was based on flow of models, typically models are dropped/released over periods of time; so a process structured on comparing model snapshots was deemed the best approach

A new feature was implemented in JACS to provide model comparison Model Comparison – checks structure and inputs between JCL model files and JCL

scenarios within a specific Model

2017 NASA Cost and Schedule Symposium 10

JCL Model Comparison - Details

Unique

Task ID

Active

Task ID

Comparison

Task ID Active Schedule Task Names Comparison Schedule Task Names Task Type Active Comparison Match Active Comparison Delta Active Comparison Delta Active

Comparis

on Delta

230 1 1 Missile System Project Missile System Project Summary Standard Standard Yes 10/1/2013 10/1/2013 7/26/2021 8/23/2021 -20 Days 2040 2060 -20

329 2 2 Contract Award Delay Contract Award Delay Summary Standard Standard Yes 10/1/2013 10/1/2013 12/2/2013 12/2/2013 45 45

330 3 3 Planned Award Planned Award Milestone Standard Standard Yes 10/1/2013 10/1/2013 10/1/2013 10/1/2013 0 0

331 4 4 Delayed Award Delayed Award Activity Standard Standard Yes 10/1/2013 10/1/2013 12/2/2013 12/2/2013 45 45

284 5 5 Contract Award Contract Award ProgramEvent Standard Standard Yes 12/2/2013 12/2/2013 12/2/2013 12/2/2013 0 0

280 6 6 Milestones Milestones Summary Standard Standard Yes 8/4/2014 8/4/2014 5/16/2016 6/13/2016 -20 Days 465 485 -20

281 7 7 EMD SDR Milestone EMD SDR Milestone ProgramEvent Standard Standard Yes 8/4/2014 8/4/2014 8/4/2014 8/4/2014 0 0

282 8 8 EMD PDR Milestone EMD PDR Milestone ProgramEvent Standard Standard Yes 4/13/2015 5/11/2015 -20 Days 4/13/2015 5/11/2015 -20 Days 0 0

283 9 9 EMD CDR Milestone EMD CDR Milestone ProgramEvent Standard Standard Yes 5/16/2016 6/13/2016 -20 Days 5/16/2016 6/13/2016 -20 Days 0 0

190 10 10 EMD Phase EMD Phase Summary Standard Standard Yes 12/2/2013 12/2/2013 1/15/2018 2/12/2018 -20 Days 1075 1095 -20

2 11 11 EMD Hardware EMD Hardware Summary Standard Standard Yes 12/3/2013 12/3/2013 8/7/2017 9/4/2017 -20 Days 960 980 -20

16 12 12 EMD HW System Design EMD HW System Design Activity Standard Standard Yes 12/3/2013 12/3/2013 8/4/2014 8/4/2014 175 175

15 13 13 EMD HW Initial Design EMD HW Initial Design Activity Standard Standard Yes 8/5/2014 8/5/2014 4/13/2015 5/11/2015 -20 Days 180 200 -20

14 14 14 EMD HW Detailed Design EMD HW Detailed Design Activity Standard Standard Yes 4/14/2015 5/12/2015 -20 Days 5/16/2016 6/13/2016 -20 Days 285 285

13 15 15 EMD HW Final Design EMD HW Final Design Activity Standard Standard Yes 5/17/2016 6/14/2016 -20 Days 8/7/2017 9/4/2017 -20 Days 320 320

286 16 n/a EMD HW Malfunction (RR#1) no match RiskEvent

Start Finish Duration (Days)Calendar

DETAILS

Model Comparison—Example

Feature compares

Tasks

Calendars

Dates

Duration

Logic

Costs

Distributions

Risks

112017 NASA Cost and Schedule Symposium

JCL Model Comparison - Summary

Report Date 8/7/2017

Active Project Name: JACS Example File 02a Advanced - with Risk Factors

Comparison Project Name: JACS Example File 02a Advanced - with Risk Factors chng for model comp

Schedule Status

Description

Status Date

Current Start

Current Finish

Project Duration

Project Total Cost

Estimated Workdays Remaining

Project Percent Complete

Schedule Statistics

Description Delta Active Schedule Comparison Schedule Delta Active Schedule Comparison Schedule Delta

Total Lines in IMS 2 26 24 2 28 28

Total Activities 15 15 0 0

Total Milestones 2 11 9 2 14 14

Schedule Milestones 0 0 8 8

Risk Events 2 7 5 2 0 0

Risk Factors 4 4 0 0

Start/Finish Hammock Milestones 0 0 6 6

Active Schedule

54

15

25

8

7

4

6

Comparison Schedule

0.0%

WITH Duration Uncertainty WITHOUT Duration Uncertainty

8

5

4

52

15

23

6

$267,476.000$267,476.000

Comparison Schedule

Total

Active Schedule

9/1/2013

10/1/2013

7/26/2021

2040 Days

2040 Days

9/1/2013

10/1/2013

8/23/2021

2060 Days

2060 Days

0.0%

SUMMARY

Go To Excel File

Validating Calculations

Baseline Delta Report was added to verify that JACS calculation will return similar values* as entered into the Project/P6 model

Task NameDuration

(Project)

Duration

(JACS)

Duration Δ

(days)

Start

(Project)

Start

(JACS)

Start Δ

(days)

Finish

(Project)

Finish

(JACS)

Finish Δ

(days)

Is Critical

(Project)

Is Critical

(JACS)

Is Critical

Match

Total Slack

(Project)

Total Slack

(JACS)

Total Slack Δ

(days)

Missile System Project 2040 2039 (1) 10/1/2013 10/1/2013 0.00 7/26/2021 7/26/2021 0.00 Yes Yes Yes 0 -1 (1)

Contract Award Delay 45 45 0 10/1/2013 10/1/2013 0.00 12/2/2013 12/2/2013 0.00 Yes Yes Yes 0 0 0

Planned Award 0 0 0 10/1/2013 10/1/2013 0.00 10/1/2013 10/1/2013 0.00 Yes Yes Yes 0 0 0

Delayed Award 45 45 0 10/1/2013 10/1/2013 0.00 12/2/2013 12/2/2013 0.00 Yes Yes Yes 0 -1 (1)

Contract Award 0 0 0 12/2/2013 12/2/2013 0.00 12/2/2013 12/2/2013 0.00 Yes Yes Yes 0 -1 (1)

Milestones 465 465 0 8/4/2014 8/4/2014 0.00 5/16/2016 5/16/2016 0.00 Yes Yes Yes 0 0 0

EMD SDR Milestone 0 0 0 8/4/2014 8/4/2014 0.00 8/4/2014 8/4/2014 0.00 Yes Yes Yes 0 0 0

EMD PDR Milestone 0 0 0 4/13/2015 4/13/2015 0.00 4/13/2015 4/13/2015 0.00 Yes Yes Yes 0 0 0

EMD CDR Milestone 0 0 0 5/16/2016 5/16/2016 0.00 5/16/2016 5/16/2016 0.00 Yes Yes Yes 0 0 0

EMD Phase 1075 1075 0 12/2/2013 12/2/2013 0.00 1/15/2018 1/15/2018 0.00 Yes Yes Yes 0 0 0

EMD Hardware 960 960 0 12/3/2013 12/2/2013 0.00 8/7/2017 8/7/2017 0.00 Yes Yes Yes 0 0 0

EMD HW System Design 175 175 0 12/3/2013 12/2/2013 0.00 8/4/2014 8/4/2014 0.00 Yes Yes Yes 0 0 0

EMD HW Initial Design 180 180 0 8/5/2014 8/4/2014 (0.13) 4/13/2015 4/13/2015 0.00 Yes Yes Yes 0 0 0

EMD HW Detailed Design 285 285 0 4/14/2015 4/13/2015 (0.25) 5/16/2016 5/16/2016 0.00 Yes Yes Yes 0 0 0

EMD HW Final Design 320 320 0 5/17/2016 5/16/2016 (0.38) 8/7/2017 8/7/2017 0.00 No No Yes 115 115 0

EMD HW Malfunction (RR#1) 0 0 0 8/7/2017 8/7/2017 0.00 8/7/2017 8/7/2017 0.00 No No Yes 115 115 0

EMD Software 1075 1075 0 12/3/2013 12/2/2013 0.00 1/15/2018 1/15/2018 0.00 Yes Yes Yes 0 0 0

EMD SW System Design 175 175 0 12/3/2013 12/2/2013 0.00 8/4/2014 8/4/2014 0.00 Yes Yes Yes 0 0 0

EMD SW Initial Design 180 180 0 8/5/2014 8/4/2014 (0.13) 4/13/2015 4/13/2015 0.00 Yes Yes Yes 0 0 0

2017 NASA Cost and Schedule Symposium

* Note: JACS calculation engine recreates the schedule logic internally in order to quickly and efficiently run the CPM (critical path method) engine

12

Results Insight

The ability to quickly access and understand the model results is a significant strength of JACS and a key need for JCL analysts

Several New Features were implemented in JACS to provide rapid access for JCL analysts

Critical Path Report– Provides insight into the deterministic critical path and the probabilistic

critical pathEnables identification of tasks that are hidden drivers (on probabilistic and not on deterministic) or are non-drivers (on deterministic and not on probabilistic)

–

Active Risk Impact– Provides insight into true impact of how a risk will affect results when it occurs

This focuses only on the conditional scenarios when the risk is active and gives better insight into how much of a potential driver a risk can be if it’s likelihood were to increase

–

Enhanced Insight Viewer– Additional reports were added to the Insight Viewer and also the ability to

specifically identify a task or summary to evaluate

132017 NASA Cost and Schedule Symposium

Results Insight – Critical Path Analysis

14

ACTIVITIES

RISKS

2017 NASA Cost and Schedule Symposium

Critical Path Analysis Report for ActivitiesSchedule: JACS Example File 02a Advanced - with Risk Factors.mpp

Report Date: 8/7/2017

Task ID Activity Name

Unique

ID Risk ID Start

Mean

Duration On Plan CP Criticality Duration Uncertainty Successors Predecessors

4 Delayed Award 331 10/1/2013 55.06 Yes 81% Tri(0,45,120,0,100) 5 3

12 EMD HW System Design 16 12/2/2013 196.73 Yes 46% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)7,27,52 5

18 EMD SW System Design 78 12/2/2013 199.91 Yes 39% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)7,28,53 5

13 EMD HW Initial Design 15 8/4/2014 202.23 Yes 44% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)8,30,52 7

19 EMD SW Initial Design 83 8/4/2014 205.63 Yes 39% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)8,31,53 7

26 EMD System Design Multi-Impact Risk (RR#3) 289 EMD_SysDes_Multi 8/4/2014 42.23 Yes 89%

27 HW System Design Risk 290 HWSysDes3.1 8/4/2014 4.40 Yes 52% Tri(15,25,45,10,90);Correl(SYSDESRISK=0.7)7 12

28 SW System Design Risk 291 SWSysDes3.2 8/4/2014 5.12 Yes 45% Tri(20,30,50,10,90);Correl(SYSDESRISK=0.7)7 18

14 EMD HW Detailed Design 14 4/13/2015 343.02 Yes 33% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)9,51,52 8

20 EMD SW Detailed Design 88 4/13/2015 348.68 Yes 49% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)9,21,51,53 8

29 EMD Initial Design Multi-Impact Risk (RR#4) 292 EMD_InitDes_Multi4/13/2015 45.25 Yes 76%

30 HW Initial Design Risk 293 HWInitDes4.1 4/13/2015 5.85 Yes 32% Tri(15,25,45,10,90);Correl(INITDESRISK=0.7)8 13

31 SW Initial Design Risk 294 SWInitDes4.2 4/13/2015 6.69 Yes 45% Tri(20,30,50,10,90);Correl(INITDESRISK=0.7)8 19

15 EMD HW Final Design 13 5/16/2016 384.86 No 18% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)32,25FF,16,51,52 9

21 EMD SW Design Failure (RR#2 ) 287 SW_RR#2 5/16/2016 39.18 Yes 75% Tri(80,120,180,10,90) 9 20

22 EMD SW Refinements 110 5/16/2016 532.50 Yes 63% Tri*(90,100,120,10,90);Correl(EMDDURATION=0.5)32,25FF,51,53 9

16 EMD HW Malfunction (RR#1) 286 HW_RR#1 8/7/2017 34.86 No 63% Tri(140,160,220,10,90) 32,25FF 15

36 Prod Procure Materials 313 1/15/2018 643.67 Yes 81% Tri*(85,100,125,10,90);Correl(PRODDURATION=0.7)37SS+140Days,49,54 34

37 Prod Air Vehicle 314 7/30/2018 909.57 Yes 81% Tri*(85,100,125,10,90);Correl(PRODDURATION=0.7)38SS+80Days,49,48FF,5436SS+140Days

38 Prod IAT&C 316 11/19/2018 625.86 Yes 40% Tri*(85,100,125,10,90);Correl(PRODDURATION=0.7)41SS,42SS,40SS+180Days,39,48FF,45FF+45Days37SS+80Days

41 Prod Common Support Equipment 319 11/19/2018 341.58 No 0% Tri*(70,105,140,10,90);Correl(PRODDURATION=0.7)49,54 38SS

42 Prod Peculiar Support Equipment 320 11/19/2018 292.75 No 0% Tri*(70,105,140,10,90);Correl(PRODDURATION=0.7)49,54 38SS

40 Prod ST&E 317 7/29/2019 546.06 Yes 40% Tri*(70,105,140,10,90);Correl(PRODDURATION=0.7)48FF,49 38SS+180Days

39 Air Vehicle Test Failure RR#5 332 AV_RR#5 3/8/2021 14.88 No 0% Tri(80,140,220,10,90) 48FF,45FF+45Days 38

Critical Path Analysis Report for RisksSchedule: JACS Example File 02a Advanced - with Risk Factors.mpp

Report Date: #######

Activated Task ID Risk Name

Unique

ID Risk ID Start On Plan CP Criticality Likelihood

Conditional

Criticality Mean Duration Mean TI Cost Mean TD Cost Successors Predecessors

Yes 26 EMD System Design Multi-Impact Risk (RR#3) 289 EMD_SysDes_Multi

Yes 27 HW System Design Risk 290 HWSysDes3.1 8/4/2014 Yes 7.80% 15.00% 52.00% 29.00 $543.00 $0.00 7 12

Yes 28 SW System Design Risk 291 SWSysDes3.2 8/4/2014 Yes 6.80% 15.00% 45.33% 34.00 $888.00 $0.00 7 18

Yes 29 EMD Initial Design Multi-Impact Risk (RR#4) 292 EMD_InitDes_Multi

Yes 30 HW Initial Design Risk 293 HWInitDes4.1 4/13/2015 Yes 6.40% 20.00% 32.00% 29.00 $582.00 $0.00 8 13

Yes 31 SW Initial Design Risk 294 SWInitDes4.2 4/13/2015 Yes 9.00% 20.00% 45.00% 33.00 $1,006.00 $0.00 8 19

Yes 21 EMD SW Design Failure (RR#2 ) 287 SW_RR#2 5/16/2016 Yes 22.60% 30.00% 75.33% 131.00 $3,564.00 $0.00 9 20

Yes 16 EMD HW Malfunction (RR#1) 286 HW_RR#1 8/7/2017 No 12.60% 20.00% 63.00% 174.00 $1,041.00 $0.00 32,25FF 15

Yes 39 Air Vehicle Test Failure RR#5 332 AV_RR#5 3/8/2021 No 0.00% 10.00% 0.00% 149.00 $11,191.00 $0.00 48FF,45FF+45Days 38

When Risk is Active

Results Insight – Active Risk Impact

2017 NASA Cost and Schedule Symposium

Risk Event RankingSchedule: JACS Example File 02a Advanced - with Risk Factors.mpp

Report Date:8/7/2017

Ranking

Activated Risk Description Task ID Risk IDActivity

Impacted

Based on

ActiveOverall Active Likelihood Criticality

Conditional

Criticality

Overall Mean

Duration

Active Mean

Duration

CoC Mean

Duration

Overall CoC

Duration Impact

Active CoC

Duration Impact

Yes EMD HW Malfunction (RR#1) 16 HW_RR#1 15 1 35.9% 41.5% 20.0% 12.6% 63.0% 34.9 174.3 174.9 22.0 110.2

Yes EMD SW Design Failure (RR#2 ) 21 SW_RR#2 20 2 48.4% 37.3% 30.0% 22.6% 75.3% 39.2 130.6 131.6 29.7 99.1

Yes EMD System Design Multi-Impact Risk (RR#3) 26 EMD_SysDes_Multi

Yes HW System Design Risk 27 HWSysDes3.1 12 4 3.8% 5.8% 15.0% 7.8% 52.0% 4.4 29.3 29.8 2.3 15.5

Yes SW System Design Risk 28 SWSysDes3.2 18 3 3.9% 6.1% 15.0% 6.8% 45.3% 5.1 34.1 35.5 2.4 16.1

Yes EMD Initial Design Multi-Impact Risk (RR#4) 29 EMD_InitDes_Multi

Yes HW Initial Design Risk 30 HWInitDes4.1 13 6 3.3% 3.8% 20.0% 6.4% 32.0% 5.8 29.2 31.5 2.0 10.1

Yes SW Initial Design Risk 31 SWInitDes4.2 19 5 4.8% 5.5% 20.0% 9.0% 45.0% 6.7 33.4 32.6 2.9 14.7

Yes Air Vehicle Test Failure RR#5 39 AV_RR#5 38 7 0.0% 0.0% 10.0% 0.0% 0.0% 14.9 148.8 0.0 0.0 0.0

Information Probabilities ImpactsRelative Contribution

Ranking: Ranking of risk events based on contribution to schedule when risk is on the critical path (order of the Active Relative Contribution values)

Relative Contribution: Proportion of a risk to the sum of all risks for the Overall/Active CoC Duration Impacts

Probabilities

Likelihood: Probability risk is active (user-defined)

Criticality: Probability risk is on the critical path

Conditional Criticality: Probability risk is critical, given it is active

Impacts Overall Mean Duration: Average duration of risk, considering all iterations

Active Mean Duration: Average duration of risk, considering only those iterations when it is active

Conditional Criticality Mean Duration: Average duration of risk, considering only those iterations when it is active and lands on the critical path

Overall Conditional Criticality Duration Impact: Expected impact to the critical path considering all iterations (CoC Mean * Criticality)

Active Conditional Criticality Duration Impact: Expected impact to the critical path considering only those iterations when it lands on the critical path (CoC Mean * Conditional Criticality)

15

Compare Scenario Results

2017 NASA Cost and Schedule Symposium

A key aspect of JCL analysis is to assess scenarios, including in the review process where the SRB may adjust inputs they feel should be modified (e.g., Risk and Distributions), and assess the change

A new capability was added to Insight to allow comparison

16

JCL Model Assessment

2017 NASA Cost and Schedule Symposium

JACS Reports menu—access to tools/outputs to assess models

Note: Dashboard style reports available in JACS – Insight analysis tool

Assesses both the schedule and JACS specific entries and generates a report to help you find and correct deviations from best practices

Summarizes and outlines tasks impacted by risk factors

Summarizes every aspect of the contents of the schedule at a high level as well as a detailed breakdown for each row

Compares the values JACS calculates without uncertainty (often referred to as the plan values or the point estimate values) to the values Microsoft Project calculates

Provides a comprehensive output of all the costs on all the rows in the model broken out by FY

Compares 2 JACS files in great detail—Flags any differences in inputs, outputs, and attributes

Reveals critical information regarding the effects of risk events—metrics include conditional criticality

Provides an overview of each Task and Risk Event’s relationship to the critical path individually that includes key metrics

Classic risk statistics report with filter options for duration & cost by task

17

JACS is available to all NASA users

JACS is included within the ACEIT software suite NASA has volume licensing for nine (9) NASA centers

NASA Civil Servants and in-situ NASA Contractors have fullaccess to ACEIT and JACS

JACS software is available for download fromwww.oncedata.com

One of the most frequent downloads

JACS technical support is fully covered by the ACEIT license Contact ACEIT_Support@tecolote.com to enter a help ticket

Formal JACS Training Course is available Has previously been held at NASA HQ, NASA LaRC, and NASA JSC

2017 NASA Cost and Schedule Symposium 18

Summary

Developing a sound JCL model is a challenging task involving many teams, inputs, and moving parts

JACS’ analytical tools help assess Schedule Structure, Cost, Risk, and Uncertainty Inputs during JCL modeling process

Exploiting these tools/reports helps ensure the JCL model best simulates the program/project

2017 NASA Cost and Schedule Symposium 19

THANK YOU

2017 NASA Cost and Schedule Symposium 20