Mission Operations Cost Estimation Tool (MOCET)

2019 UpdatesMarc Hayhurst, Elliott Tibor, Brian Wood

The Aerospace Corporation

Cindy Daniels, Lissa Jordin, Washito Sasamoto, Waldo RodriguezNASA Langley Research Center

Science Office for Mission Assessments (SOMA)

August 14, 2019

Approved for public release OTR 2019-01048.

© 2019 The Aerospace CorporationAerospace acknowledges NASA’s sponsorship under Contract NNL11AA01B Task Order 80LARC18F0021

2

Topics

• MOCET Overview & Status

• Updates in MOCET v1.4

• Planned Reserves Process

• User Community

• Summary & Future Work

3

MOCET Overview

• The Mission Operations Cost Estimation Tool (MOCET) – A capability for Phase E estimation jointly developed by The Aerospace Corporation and NASA

Science Office for Mission Assessments (SOMA)– Based on actual costs of historical missions with emphasis on competed missions– Constructed by breaking the mission operations cost into the various phases– Has few subjective inputs– Estimates total Phase E mission cost– Implemented entirely in Excel and requires no additional software or tools– Also includes a user manual which provides additional instruction and background

Mar-34 Oct-34 Apr-35 Nov-35 May-36 Dec-36 Jun-37

Cruise

Approach/EDL

First Landed Month

Landed Prime Operations

Mars 2034 RoverMission Schedule

Download from ONCE Model Portal https://oncedata.msfc.nasa.gov

Available external to NASA via https://software.nasa.gov

For more information:Email: MOCET@aero.org

4

Bold = New Data in v1.4 Potential New Data for possible version 1.X update

MOCET Mission Types Database Overview

Mission/CER Type Program Missions

Planetary

Discovery MESSENGER, Stardust, Deep Impact, GRAIL, NEAR, Dawn

Mars Scout PhoenixRobotic Lunar Exploration LRO, LADEENew Frontiers New Horizons, Juno (Orbital Ops), OSIRIS-REx (Cruise, Approach/Orbit Insertion)Mars Exploration MRO, Odyssey, MER, MSL, MAVEN, Insight

Earth ScienceEarth System Science Pathfinder (ESSP) GRACE, CloudSat, CALIPSO, Aquarius, OCO-2, CYGNSS, OCO-3, GEDI

Earth Systematic Missions (ESM) Aqua, Aura, Terra, Jason-1, OSTM, ICESat, GPM, SMAP, TSIS-1, ICESat-2, GRACE-FO

Explorers

Mission of Opportunity (MO) Suazku (ASTRO-E2), TWINS, CINDI, NICER, GOLD

Small Explorers (SMEX) NuSTAR, IRIS, IBEX, AIM, GALEX, RHESSI

Medium Explorers (MIDEX) THEMIS, Swift, WISE, TESS

Near Earth Discovery Helio-Astro

Discovery Genesis, KeplerSolar Terrestrial Probes (STP) STEREO, TIMED, MMSLiving With a Star (LWS) RBSP, SDO, PSPCosmic Origins Spitzer

Physics of the Cosmos Fermi, Chandra

Data Sources Utilized and Consulted

– NASA SAP BusinessWarehouse• Monthly Expenditures

– Monthly Flight Project Reviews (FPR)and Monthly Status Reviews (MSR)• Monthly Expenditures, Mission Events

& Schedule

– NASA Cost Analysis DataRequirements (CADRe)• Project development costs used

to assign mission class

5

MOCET CER UpdatesPlanetary

CER Name New Data Updated Inflation

Comments

Nominal/Quiescent Cruise CER X X OSIRIS-REx data incorporated

Checkout Cruise CER X Landed Missions Broken Out

Mars Landed Cruise CER X X New CER with Phoenix, Insight, and MSL

Orbital Operations CER X X Juno & OSIRIS-REx data incorporated.

Flyby/Encounter CER X

Approach/Orbit Insertion/Reduction CER X X OSIRIS-REx data incorporated

Approach EDL CER X X Insight data incorporated

Landed Checkout CER X X Renamed; Insight data incorporated

Landed Prime Operations X X Insight data incorporated

Data/Sample Analysis & Archiving X

Extended Orbital Ops CER X

Extended Landed Ops CER X

Earth/Near-Earth/OtherCER Name New Data Updated

InflationComments

Earth/Near-Earth Orbiting Checkout CER X X

Earth Science Prime Operations CER X X CYGNSS, SMAP

Explorer Prime Operations CER X X NICER, GOLD, TESS data incorporated

Near Earth Discovery Helio Astro Prime Operations CER X

Instrument Only Operations CER X X New CER, Satellite and ISS Hosted

6

Explorers Prime Operations Phase CER• CER Function:𝑌𝑌 = 0.5338 + 0.3740 ∗ 𝐼𝐼𝐼𝐼𝐼𝐼 − 0.3583 ∗ 𝐼𝐼𝐼𝐼𝐼𝐼 + 0.1225 ∗ 𝐼𝐼𝐼𝐼

Variable Name Units Definition

Y Cost FY18 $M Average monthly checkout phase cost

IMe IsMedium? 0 or 1 0 No, 1 Yes, Is this a medium mission class mission?

IMi IsMicro? 0 or 1 0 No, 1 Yes, Is this a micro mission class mission?

IA IsAstro? 0 or 1 0 No, 1 Yes, Is this an astrophysics mission?

• Goodness of FitMeasure Value

Regression Method OLSStandard Error (RSS) 0.07

Average Percentage Bias 0.0%Coefficient of Determination (R^2) 0.96

Adjusted R^2 0.95Pearson's Correlation Sqd (r^2) 0.96

Number of Observations 15Number of Input Variables 3

$1.2

$1.0

$0.8

edtc $0.6

edi

rP $0.4

$0.2

$- $- $0.2 $0.4 $0.6 $0.8 $1.0 $1.2

Actual

• Database

1. IBEX (IMe 0, IMi 0, IA 0) 6. Swift (IMe 1, IMi 0, IA 1) 11. Suzaku (IMe 0, IMi 1, IA 1)

2. AIM (IMe 0, IMi 0, IA 0) 7. WISE (IMe 1, IMi 0, IA 1) 12. IRIS (IMe 0, IMi 0, IA 0)*

3. NuSTAR (IMe 0, IMi 0, IA 1) 8. RHESSI (IMe 0, IMi 0, IA 0) 13. NICER (IMe 0, IMi 1, IA 1)

4. GALEX (IMe 0, IMi 0, IA 1) 9. CINDI (IMe 0, IMi 1, IA 0) 14. TESS (IMe 1, IMi 0, IA 1)

5. THEMIS (IMe 1, IMi 0, IA 0) 10. TWINS (IMe 0, IMi 1, IA 0) 15 GOLD (IMe 0, IMi 1, IA 0)

Additional Notes• *IRIS was a SMEX mission but the development cost grew beyond small mission class boundary. In operations the IRIS average monthly cost is also higher than any other SMEX mission.• **Copies of identical instruments (THEMIS, TWINS) or suites of similarly functioned sensors (CINDI) are considered to be a single instrument since the same data processing tools can be used for them.• ***Earth & near Earth orbiting multiple spacecraft missions such as THEMIS may be estimated with MOCET without further adjusting the output estimate.

7

Mars EDL Cruise Prime Phase CER• CER Function: 𝑌𝑌 = 0.0056 ∗ 𝑅𝑅𝑅𝑅𝐼𝐼1.1388

Variable Name Units Definition

Y Cost FY19 $M Planetary mission average monthly prime checkout cruise phase cost

RLM Rover/Lander Mass kg > 0Mass of rover or lander plus payload instruments for which NASA will fund operations (exclude contributed instruments).

• Goodness of FitMeasure Value

Regression Method GERM ZMPEStandard Error of the Estimate 14.2%

Average Percentage Bias 0.0%Pearson's Correlation Sqd (R^2) 0.98

Number of Observations 3Number of Input Variables 1

$16

$14

$12

ed $10

tc $8

redi

P $6

$4

$2

$- $- $2 $4 $6 $8 $10 $12 $14 $16

Actual

• Database

1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6)

4. MER (A/B)*

Additional Notes• *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support

estimation of planetary multiple spacecraft missions

8

Landed Operations Prime Phase CER• CER Function: 𝑌𝑌 = 0.0591 ∗ 𝑅𝑅𝑅𝑅𝐼𝐼0.7034

Variable Name Units Definition

Y Cost FY19 $M Planetary mission average monthly prime landed operations phase cost

RLM Rover/Lander Mass kg > 0Mass of rover or lander plus payload instruments for which NASA will fund operations (exclude contributed instruments).

• Goodness of FitMeasure Value

Regression Method GERM ZMPEStandard Error of the Estimate 7.2%

Average Percentage Bias 0.0%Pearson's Correlation Sqd (R^2) 0.98

Number of Observations 3Number of Input Variables 1

$8

$7

$6

ed $5

tc $4

redi

P $3

$2

$1

$- $- $1 $2 $3 $4 $5 $6 $7 $8

Actual

• Database

1. Phoenix (342.9) 2. MSL (919.9) 3. Insight (318.6)

4. MER (A/B)*

Additional Notes• *MER was not included in the model due using multiple spacecraft. At this time there is not sufficient data to recommend an adjustment for multiple planetary spacecraft and the model does not support

estimation of planetary multiple spacecraft missions

9

MOCET Reserves Approach (1 of 2)

• When calculating the reserve level for operations phase estimates, a percent reserve of the base operations budget is often considered– Rule of thumb used by some considers a minimum 10% up to 25% for Phase E– Aerospace publication* showed that on average from launch estimate to final actual

growth was around 10%

• Not a simple matter of just looking at the offeror’s proposal and seeing if they are carrying between 10% and 25% reserve on their estimate– Offerors’ estimates and MOCET estimates vary significantly– Requires a new method of computing reserves for each MOCET estimate

• To determine best method of computing reserves for MOCET– Obtained data from available Step 1 and Step 2 proposals as well as other milestones

for comparison– Generated MOCET estimates using input parameters as given at Step 1, Step 2, etc.– Also, collected the offerors estimates for comparison

*Phase E Cost Analysis for NASA Science Missions, Robert Bitten, Marc Hayhurst, Debra Emmons, Claude Freaner, and Voleak Roeum, AIAA SPACE 2012 Conference & Exposition. Pasadena, California.

10

MOCET Reserves Approach (2 of 2)

• With the MOCET estimates generated at each milestone the percent difference from the final actual was computed for each mission– The following equation was used to compute the percent difference

𝐹𝐹𝐼𝐼𝐹𝐹𝐹𝐹𝐹𝐹 𝐼𝐼𝐴𝐴𝐴𝐴𝐴𝐴𝐹𝐹𝐹𝐹 𝐶𝐶𝐶𝐶𝐶𝐶𝐴𝐴 − 𝐼𝐼𝑀𝑀𝐶𝐶𝑀𝑀𝑀𝑀 𝑀𝑀𝐶𝐶𝐴𝐴𝐼𝐼𝐸𝐸𝐹𝐹𝐴𝐴𝐼𝐼 @ 𝐼𝐼𝐼𝐼𝐹𝐹𝐼𝐼𝐶𝐶𝐴𝐴𝐶𝐶𝐹𝐹𝐼𝐼𝐼𝐼𝑀𝑀𝐶𝐶𝑀𝑀𝑀𝑀 𝑀𝑀𝐶𝐶𝐴𝐴𝐼𝐼𝐸𝐸𝐹𝐹𝐴𝐴𝐼𝐼 @ 𝐼𝐼𝐼𝐼𝐹𝐹𝐼𝐼𝐶𝐶𝐴𝐴𝐶𝐶𝐹𝐹𝐼𝐼 ∗ 100% = 𝑃𝑃𝐼𝐼𝑃𝑃𝐴𝐴𝐼𝐼𝐹𝐹𝐴𝐴 𝐷𝐷𝐼𝐼𝐷𝐷𝐷𝐷𝐼𝐼𝑃𝑃𝐼𝐼𝐹𝐹𝐴𝐴𝐼𝐼

• Then the overall average percent difference and standard deviation were computed

• Various methods to compute reserve level were attempted and these metrics were compared at each milestone to determine which methods were most effective– Methodologies driven by historical data were considered to maintain objectivity

• Concluded that the best performing method is to employ a triangle distribution with the Formal Risk (FRISK) process driven by historical percent differences of similar missions at a particular milestone– Similar missions are determined by mission category

11

Percent Difference for all Explorer Missions

• As compared to Explorer Project estimates, average percent difference for MOCET is significantly less– Interesting to note that on average Step 1 estimates were better than those at Step 2

Project Estimate % Difference (Including Reserves)250%

Mission 1

Mission 2

Mission 3200% Mission 4

Mission 5

Mission 6150% Mission 7

e Mission 8

enc

er Average

ff D

i 100%

e ent

c 72%t 71%

a erm Pit 48%

es 50% 34%

nder 24%

U 0%

te 0%

amtisere

Ov -50%

Proposal CSR PDR CDR Launch EOM

MOCET Estimate % Difference (Without Reserve)250%

Mission 1

Mission 2

Mission 3200% Mission 4

Mission 5

Mission 6150% Mission 7

e Mission 8

enc

er Average

ff D

i 100%

e ent

t ca erm Pites 50%

nder

U 10%6% 9% 9% 8% 2%

te 0%

amtisere

Ov -50%

Proposal CSR PDR CDR Launch EOM

12

Percent Difference for all Planetary Missions

• As compared to Planetary Project estimates, average percent difference for MOCET is significantly less– Interesting to note that on average Step 1 estimates were better than those at Step 2

Project Estimate % Difference (Including Reserves)

Mission 1

Mission 280%Mission 3

Mission 4

Mission 560%50% Average

44%40%

e 40%

enc 32%

et era ffm

ii Dt

es ent 20%

10%

nder c

erPU 0%

0%

teamtisere -20%

Ov

-40%Proposal CSR PDR CDR Launch EOM

MOCET Estimate % Difference (Without Reserve)

Mission 1

Mission 280%Mission 3

Mission 4

Mission 560%Average

een

ec 40%

e erfta fim D it 20%

ent 14%

es 9% 10%nd

erc

er 6%

P 3%U -2%

0%

teamtiser -20%

eO

v

-40%Proposal CSR PDR CDR Launch EOM

13

MOCET Reserves Process Overview

Cost Estimates

LM

H

Note: Most-

Likely (M) is average

of all estimates

Triangular Distribution of Cost Estimates

ytilbi L M H

oba

rP

CostLower Most-Likely Upper Limit Limit

Example Cost Distribution• The Formal Risk (FRISK) process driven by

historical percent differences of similar missions at a particular milestone

• Similar missions are determined by mission category

• Adjusted mission estimates are fed into a triangular distribution

• Triangular distribution is fed into FRISK to produce an s-curve and determine the appropriate reserve level

70th Percentile

UFE Estimate

MOCET Estimate

14

ONCE Downloads and Users• To date MOCET has been downloaded from ONCE 183* times since the initial release

• *Downloads include those from inactive users and duplicate downloads – ONCE output generally only shows active users and unique downloads

• v1.0 30 downloads; v1.1 60 downloads; v1.2 34 downloads; v1.3 59 downloads as of June 2019• As of June 2019, 86 unique users have downloaded MOCET from ONCE

• Since the release of v1.0 the number of users has increased steadily

0

20

40

60

80

100

120

140

160

180

200

10/6/2015

1/4/2016

4/3/2016

7/2/2016

9/30/2016

12/29/2016

3/29/2017

6/27/2017

9/25/2017

12/24/2017

3/24/2018

6/22/2018

9/20/2018

12/19/2018

3/19/2019

6/17/2019

Cumulative Downloads

v 1.0 v 1.1NCS 2016

v 1.2NCS 2017

v 1.3NCSS 2018 v 1.0 v 1.1 NCS 2016 v 1.2 NCS 2017 v 1.3 NCSS 2018

Downloads by Month

18 17

16

1412

12 11 1110

10 9 9

8 7 7 7 76

6 5 5 5 5 5 54 4

4 3 3 3 3 32 2 2 2 2 2

2 1 1 1 1 1 10 0 0 0 0 0 0 0

0 O N D J F M Ap M J J Au Se O N D J F M Ap M J J Au Se O N D J F M M J Ja u ul a u a uc a a c a a ul c aAp a ulAu Se O N D J F M

c

Ap M J

t ov ec n-eb y n- - g p t ov ec n-eb y n- - g p t- - -ov ec n-eb y n- - g p t-ov eca eb ua a

- - - n- n-r-1r-1 y- 1 -1 -1 - - - r-1r-1 1 -1 -1 - - - r-1r-1 115 16 16 16 6 16 -1 -1 - - - r-1r-115 15 66 16 6 6 16 16

17 17

- 777 17 7 717 17 1718 18

-17 88 1818 8 8 818 18 1819 19

-99 1919

15

Public Release on software.nasa.gov• MOCET has been approved for public release via

software.nasa.gov– The direct link can be found here:

https://software.nasa.gov/software/LAR-18894-1

• Available to almost anyone* worldwide

• User must create an account on software.nasa.gov

• Fill out the request form

• *A signature authority must be provided for those representing businesses– Cannot be an engineer or those with similar titles– An authorized signature authority is considered

• president of the company• general counsel• or contract officer

• If representing self as a private individual, an additional signature is not required

• Rules may be different for other user types, Academic etc.

• Contact MOCET help staff at MOCET@aero.org with any issues you may encounter

16

software.nasa.gov Downloads and Users• There are MOCET 31 users on software.nasa.gov since its release there in April of 2017

• Most users are from Industry (14) and from NASA (7)• There are also 5 Representing Self, 3 Other Civil, and 2 Academic• Of the 31 users, 11 are also international coming from Africa, Asia, Australia, Europe, and South America

17

Summary & Next Steps

• New data is being added to the model which will be released as MOCET v1.4

• MOCET will continue to be periodically updated with new mission data

• Model is currently being used by both evaluators and proposers

• We will continue to engage and grow the user community

Download MOCETONCE Model Portal https://oncedata.hq.nasa.govAvailable external to NASA via https://software.nasa.gov

For More InformationDemo session (see agenda for time and room)Email: MOCET@aero.org

Publications• Mission Operations Cost Estimation Tool (MOCET) Version 1.3 and Beyond, 2019 IEEE Aerospace Conference, Big Sky, MT• The Mission Operations Cost Estimation Tool (MOCET): Development History and 2018 Updates, 2018 AIAA SPACE and Astronautics Forum and Exposition. Orlando FL• Mission Operations Cost Estimation Tool (MOCET) v1.3, 2018 NASA Cost and Schedule Symposium, August 2018, Greenbelt MD• Mission Operations Cost Estimation Tool (MOCET), 2017 IEEE Aerospace Conference, Big Sky, MT• Mission Operations Cost Estimation Tool (MOCET) FY17 Update, 2017 NASA Cost and Schedule Symposium, August 2017, Washington DC• Mission Operations Cost Estimation Tool (MOCET) Update, 2016 NASA Cost Symposium, August 2016, Cleveland OH• Mission Operations Cost Estimation Tool (MOCET), 2015 NASA Cost Symposium, August 2015, Mountain View CA