View
4
Download
0
Category
Preview:
Citation preview
Distribution Statement A: Approved for public release. Distribution is unlimited..
https://www.CTO.mil @DoDCTO
Distribution Statement A: Approved for public release. Distribution is unlimited..
The Developmental Evaluation Framework and its Effect on Test and Evaluation of a Major Defense Space Acquisition
Mr. Rick ThomasDeputy Director for Space Systems, Developmental Test, Evaluation and Assessments Office of the Deputy Director, Engineering, OUSD (R&E)
37th International Test and Evaluation SymposiumSeptember 16, 2020
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Why We Are Here?
Programs benefit by linking decisions, evaluation, test, resources, and schedule to yield performance information of system capability
The Developmental Test, Evaluation and Assessments (DTE&A) office in OSD created the Developmental Evaluation Framework (DEF) as a planning and execution tool to assist a program with its test and evaluation (T&E) strategy
2
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Purpose
Present the value and impact the DEF has had on a space program to deliver Warfighter capability
Describe how the program manager implemented the DEF as a planning and execution tool to assist the program in implementing an effective T&E strategy
Provide lessons learned from the Space Fence experience
3
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Space Fence Ground-Based Radar System
Space Fence is a sophisticated system to dramatically improve the way U.S. Space Force identifies and tracks objects in space.
4
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Space Fence DT&E Background
Starting in 2012, worked closely with Space Fence program office in producing the first DEF to support the program’s Engineering, Manufacturing, Development (EMD) acquisition phase
– Created DEF and included it in Request for Proposal (RFP)
– Vendor leaders stated the DEF in the RFP let them know what the Government needed, resulting in lower bids from vendors
– Bids were approximately 18-20 percent below Government expectations
– The DEF was incorporated into section 3.2 of the Test and Evaluation Master Plan (TEMP) in support of the Milestone B decision
5
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
What the DEF Is and Does
The DEF guides development of the DT&E strategy by:
– Focusing the thought process on logically identifying critical program decisions
– Defining the information needed to inform the decisions– Defining the test and modeling and simulation (M&S) events
needed to generate data, which when evaluated provides the knowledge
The DEF design calls for the PM to identify key data for assessing progress toward achieving developmental evaluation objectives.
6
Distribution Statement A: Approved for public release. Distribution is unlimited..
DEF ImplementationTechnology
Maturation &Risk Reduction
Engineering &ManufacturingDevelopment
Production& Deployment
Operations& Support
MaterielSolutionAnalysis
A B CCDD CPD
ValidatedUser NeedsStatement
MDDICD FRP
PM developed DEF to support EMD RFP
USD(R&E)Tech Modernization
Activity Focus
D(DTE&A) Span ofEngagement
Vendors provided DEFs in their EMD
proposal
DEF guided DT&E through integrated
testing
Developmental Evaluation Framework is a living document** Acronyms in Notes
7
Expanding DEF Applicationrapid prototyping & fielding,
mission engineering; enterprise architecture
DEF OriginsDeveloped to assist DoDI
5000.02 ACAT 1 programs
DEF DevelopmentThought process to scope and align an evaluation strategy to
inform decision-making
1.
2.
3.
Distribution Statement A: Approved for public release. Distribution is unlimited..
DEF Methodology
8
Decisions
Evaluation
Test / M&S
Resources
Schedule
Def
ine Inform
Def
ine Data
ExecuteDef
ine
Design of Experiments-Based
Test Design
Define
Plan
- Performance- Interoperability- Cybersecurity- Reliability
Distribution Statement A: Approved for public release. Distribution is unlimited..
Developmental DEF Evolution
9
The Program Office modified the DEF over time to address important changes:– More mature cooperation with the vendor and Government
management and updates to decision points
– The addition of cybersecurity
During this period, the 45th and 47th Test Squadrons stood up as Lead Developmental Test Organizations (LDTO) and learned to execute several Developmental Test Risk Reduction events
The test results of these events were returned to the Program Office and folded into Lockheed Martin’s (LM) monthly metric reporting
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Integrated T&E Touch Points
10
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Space Fence Reporting
Leadership Team utilized DEF as a tool to identify technical performance metrics required to achieve and retire developmental objectives
– LM: Reported system technical performance measures (TPMs) and provided copies to Government
– Government Program Manager: Reported system TPMs quarterly in Defense Acquisition Executive Summery
– Air Force 46th Test Squadron: Reported status of DEF evaluation measures after contractor and final Government DT&E events
11
Distribution Statement A: Approved for public release. Distribution is unlimited..
Example of Score Card
12
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
DT&E DEF Retrospective
Space Fence DEF
– The winning contractor stated the DEF lets the contractor know what the government wants, resulting in his vendor’s lower bid (ICOTE,); both vendors bids were approximately 18-20 percent below Government expectations
Government PM lauded the DEF Core Team efforts
– Stated the DEF was a solid and usable product for evaluating design and development as well as system capability (performance)
13
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Scorecard Lessons Learned
The scorecard, at every level of leadership (contractor, PM, engineers, and DTE&A), provides a quick glance of how the critical performance metrics were satisfied to show progress-to-plan
The Space Fence scorecard was a solid tool for tracking critical technical performance metrics
By highlighting issues in red, the scorecard easily presented to the vendor and Government developers what needed to be focused on and fixed
Many other programs would benefit from program implementation
14
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Upcoming DEF Adaptations
DEF can be adapted to use through different acquisition strategies and pathways:– Adaptive Acquisition Framework – Rapid Acquisition/Prototyping– Software (DevSecOps) – Mission Engineering– Enterprise Architecting
Evolving concept of DEF and Operational Evaluation Framework (OEF) as foundations for a Unified Evaluation Framework (UEF) to increase focus on full lifecycle decision-support, as well as Digital Engineering and Mission-Based Systems Engineering (MBSE) linkage
15
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Summary
DEFs represent the intersection of program management, systems engineering, and contractor T&E and DT&E, providing a strong foundation for integrating test efforts
DEF has been applied to over 100 programs supporting major program decisions
Programs like SF and its DEF, from inception to RFP, to vendor selection through to OT&E and IOC, enable timely delivery of capabilities with better buying power
SF program delivered on time, meeting its threshold date for its final Acquisition Program Baseline milestone
DEF served as an effective DT&E tool for the PM and contractor to deliver a fully capable system to Warfighters
16
Distribution Statement A: Approved for public release. Distribution is unlimited..Distribution Statement A: Approved for public release. Distribution is unlimited..
Acronyms• Capability Development Document (CDD)• Capability Production Document (CPD)• Developmental Evaluation Framework (DEF) • Developmental Test and Evaluation (DT&E)• Director, Developmental Test Evaluation and Assessment (D,DTE&A)• Engineering, Manufacturing, Development (EMD)• Full-Rate Production (FRP)• Ground Based Strategic Deterrence (GBSD)• Initial Capabilities Document (ICD)• The Industrial Committee on Operational Test and Evaluation (ICOTE)• Initial Operational Capability (IOC)• Lockheed Martin (LM)• Materiel Development Decision (MDD)• Next Generation Overhead Persistent Infra-Red (NG OPIR)• Office of the Secretary of Defense (OSD)• Operational Test and Evaluation (OT&E)• Request for Proposal (RFP)• Test and Evaluation (T&E)• Test and Evaluation Master Plan (TEMP)• Technology Maturation & Risk Reduction (TMRR)• Technical performance measures (TPM)• Under Secretary of Defense for Research and Engineering (USD(R&E))
17
Recommended