Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
NDIA DT&E Committee2014
1
Chief Developmental Tester & Industry Test Lead – Partnering for success
NDIA DT&E Committee Project LeadJoe Manas – Raytheon Company
NDIA DT&E Committee2014
2
Agenda
• Origin of project (NDIA DT&E Committee)
• Project Overview
• Strategy to align Industry with new Chief Development Tester & Initiatives
• DASD(DT&E) Shift Left initiative
• Critical interactions and acquisition timeline
• Examples of excellence
• Recommendations
• Next Steps – Future Work
NDIA DT&E Committee2014
3
T&E Themes
ForConferences
SEDCommittee
CollaborativeProjects
DT&E Committee Structure (since 2010)
NDIA Systems Engineering Conference (Q4) Collaborative Projects
NDIA Test and Evaluation Annual Conference (Q1)Periodic T&E Committee-Focused Conferences
Cooperation with other NDIA Divisions
NDIA Industrial Committee on Test and Evaluation
(ICOTE)
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
T&E Themes & Issues
Project ResultsDT&E Themes
NDIATest and Evaluation
Division
NDIASystems Engineering
Division
NDIADT&E Committee
SE Themes & Issues
NDIA DT&E Committee2014
4
Chief Developmental Tester Project
Goal: Propose a model for industry interaction throughout the phases of a development program with an emphasis on "Shift Left".
Title 10, Section 139bThe Secretary of Defense shall require that each major defense acquisition program be supported by—‘‘(A) a chief developmental tester; and‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program.
CDT Project teamJoe Manas (Lead) – Raytheon Brendan Rhatigan – Lockheed MartinSandi Gianotas – Boeing (BTE) Steve Scukanec – Northrop GrummanTom Simms – DASD(DT&E) Paul Alfieri – DAUJoe Wascavage – NAVAIR
NDIA DT&E Committee2014
5
Opportunity for Improvement onDevelopmental Programs
DT&E Assessments recommending to proceed to IOT&E
(2012 DT&E Annual Report)
Proceed
Do Not Proceed
IOT&ESignificant IssuesNo Issues
25%
50%
Only 41% (5 of 12) programs made it to & through a successful IOT&E
Num
ber o
f Iss
ues
Rea
lized
Issue actual discoveredEarliest opportunity for discovery
Nominal cost forfault removal (NIST data) 1x 5x 16x 40x 110x
Notional
Late discoveries are considerably more costly!
Resulting in production delays
The Cost The Impact
Copyright © 2014 Boeing. All rights reserved.
NDIA DT&E Committee2014
6
Strategy To Improve
• Comprehensive DT&E activities, earlier in the development process, better inform technical, acquisition, and programmatic decisions.
• Early and often interaction and coordination between Government (Chief Developmental Tester) and Industry (Industry Test Lead) is critical in the early identification of problems, validating fixes, and ensuring technology is mature well before committing to production.• Early use of M&S, & Live/Virtual/Constructive (LVC) testing.
• Interoperability, Cybersecurity, etc.• Leverage LVC environment into flight test & field demonstrations. [mission
thread focus]• Earlier start on Reliability Growth Planning• Cybersecurity Testing – Using the Program Protection Plan
• End Goal:• “…the real benefit of an effectively structured test program is in the
cost avoidance it can provide by discovering problems as early as possible.”
(Mr. Frank Kendall, USD(AT&L), March 2013, ITEA Journal
NDIA DT&E Committee2014
7
DASD(DT&E) Initiatives
“Shift Left” (started in 2012)• Find and Fix Problems EARLY!• Three focus areas:
- Earlier Mission Context- Earlier Interoperability Testing- Earlier Cybersecurity Testing
• Currently, the initiative is also focusing on:- System performance- Reliability
Developmental Evaluation Framework• Provides the roadmap to obtain developmental data • Knowledge gained from testing provides information
for technical, programmatic, and acquisition decisions.
• Aids programs in determining how to structure a test program.
- Shows correlation/mapping test events, resources, and decisions.
DT&E FY 2013 Annual Report
Early DT&E activities inform technical, programmatic, and acquisition decisions
This is key!
NDIA DT&E Committee2014
8
DODI 5000.02 (Enclosure 4)
Test and Evaluation Master Plan (TEMP) includes a Developmental Evaluation Framework (“T&E Roadmap”)
• Knowledge gained from testing provides information for technical, programmatic, and acquisition decisions.
Developmental Evaluation Framework:– Identifies key data that contributes to assessing progress on:
− Key Performance Parameters− Critical Technical Parameters− Key System Attributes− Interoperability requirements− Cybersecurity requirements− Reliability growth− Maintainability attributes− Developmental test objectives− Others as needed
– Show the correlation/mapping between:− Test events− Key resources− Decisions supported
The TEMP is the Program Managers primary planning and management tool for all test and evaluation activities.
NDIA DT&E Committee2014
9
T&E in Acquisition Lifecycle
Full Rate ProductionDecision ReviewCA
Technology Maturation & Risk
ReductionEngineering & Manufacturing Development
Production and Deployment
O&SMaterielSolution Analysis
MDD
RequirementsDecision
Developmental RFP Decision
B
Draft
SHIFT LEFT: Includes earlier test and evaluation planning and activities to support Milestone C decision
SRR SFR CDR SVRASR PDR
TEMP TEMP TEMP TEMPDraft DT&E
Chief Developmental Tester
Industry Test Lead
EMD ContractTMRR Contract
Industry Test Lead(s)
Lead DT&E Organization
T&E WIPT
Industry Test Leads can help with early test and evaluation planning and activities
PROJECT FOCUS:Interactions between Industry and government
Most important single decision in the life
cycle… sets in motion all that follows.
The venue for frequent interaction with all key stakeholders
NDIA DT&E Committee2014
10
CDT/ITL Descriptions
Chief Developmental TesterIs a key leader in the program office, provides advice and recommendations based on test results to the program manager to inform technical, acquisition, and programmatic decisions. Chairs the T&E WIPT.
Highly qualified professional that possesses knowledge, skills, and ability to develop and execute a comprehensive and efficient test and evaluation program
Brings the technical education, test and evaluation experience, leadership, and cross-functional insight necessary to guide the technical management, program execution, and business management efforts associated with test and evaluation and defining test requirements.
Industry Test LeadIndustry counter part to the Chief Developmental Tester and is a member of the Government T&E WIPT. Currently, has many names, (Test Architect, Test Manager, Chief Validation Engineer, etc).
Highly qualified professional that possesses knowledge, skills, and ability to plan and coordinates all Contractor T&E activities during development.
Brings the technical education, test and evaluation experience, leadership, and cross-functional insight necessary to guide the contractor test program
Operates at a level above the Contractor IPTs and provides guidance across all the Contractor IPTs (similar to that of a Chief Engineer). Works with and provides feedback based on test results to design engineers or product teams during development.
T&E is a Technical Discipline
NDIA DT&E Committee2014
11
Potential Industry Organization Models
Corporate Executives
ITLs Chief EngOffice
Individual PMs
Test VP ProgramExecutives
Corporate T&E ExecutiveCorporate Executives
Chief EngOffice
Individual PMs
ProgramExecutives
ITL Part of CE Office
CEsITLs
Corporate Executives
Chief EngOffice
ProgramExecutives
ITLs at Program Level
ITLs
Individual PMs
• Currently most organizations are set up with ITL as part of the CE office (if an ITL exists)
• “Shift Left” Initiative calls more for ITL to move out of the CE office and report direct to the program manager
• T&E has most independence in the Corporate Model
• This model does not exist in most companies, transformational
NDIA DT&E Committee2014
12
DODI 5000.02
Enclosure (4): A robust DT&E program includes a number of key activities to provide the data for decision making.
The TEMP is the Program Managers primary planning and management tool for all test and evaluation activities.
DT&E ProgramChief
Developmental Tester
Contractor Testing(Industry Test Lead)
Government Testing(Lead DT&E Organization)
Enclosure 4
NDIA DT&E Committee2014
13
DT&E Program
DT&E ProgramChief Developmental
Tester
Contractor Testing(Industry Test Lead)
Government Testing(Lead DT&E Organization)
• Provides technical expertise on T&E issues • Conducts DT&E activities as directed by
the Chief Developmental Tester • Supports certification and accreditation
activities, when feasible• Assists the Chief Developmental Tester :
• Provide oversight of contractors• Reach technically
informed, objective judgments about contractor and government T&E planning and results
• Provides technical expertise on T&E issues • Plans out the strategy and implementation of the
integrated test campaign• M&S to HWIL to ground test to field test\
• Manages enterprise resources to achieve results• Works with DT&E organization on collaborations• Conducts DT&E activities, as stated in the
contract • Supports government testing, when directed• Assist the Chief Developmental Tester:
• Participate in T&E working groups• Use Evaluation Framework to guide
integrated test planning• Contribute to test strategy that yields cost
effective test event plans
T&EWIPT
NDIA DT&E Committee2014
14
INDUSTRY EXAMPLES
NDIA DT&E Committee2014
15
Industry Survey Template
Best Practice Title:
{type title here}
Program Phase:
(check one)
Technology Maturation and Risk
Reduction (TMRR)
Engineering, Manufacturing,
and Development
(EMD)
Test Strategy Themes:
(check all that apply)
Operationally Relevant Testing
STAT Methods
Used
Driving Affordability
Early T&E “Shift Left”
Transition to Milestone C
Reliability Growth
• Describe an instance on a program where the contractor test lead partnered with the government test lead (equivalent to the new Chief Developmental Tester) and there was success:
- • Name of the contractor role:
- • Why the contractor role emerged:
- • What were the benefits of having this role:
- • Who were the advocates for the role:
- • Who were the opponents of the role:
- • Recommended interaction with Chief Developmental Tester Best Practice:
-
Industry Test Lead Results:• Achieves Integrated Testing• Promotes Operationally Relevant Testing• Enables Statistical Test Optimization• Implements “Shift Left”• Improves Affordability• Focuses on Reliability
NDIA DT&E Committee2014
16
Industry Examples: Test Strategy Excellence Themes
• Shift Left Implementation• Earlier mission context in – using M&S and L/V/C concepts• More effective interoperability and cyber testing• Mission info captured with minimum number of scenarios• Promotes operationally relevant testing• Cost-effective test program execution with coordinated test activity• Risk based definition of test events• More emphasis on suitability/logistics early in test• Test planning focus on evaluation and informing decisions• Proactive work-arounds to potential problems
• Developmental Evaluation Framework• DT scenarios defined to evaluate system in mission context• Define the data defined to be collected and evaluated during the contractor, government
DT&E, and Integrated Testing• Maximize data obtained and shared
• Optimize test program and the number of test events required – use statistics to validate• Focus on capabilities, performance, cybersecurity, interoperability, and reliability to avoid
“verification by exhaustion”• Identify potential sources of data to support reliability growth curves• Identify effective techniques for cyber testing
NDIA DT&E Committee2014
17
Industry Examples:Why Industry Lead Role Emerged• Program schedule and budget challenges
• Make sure test program had no overlapping/duplicate efforts• Acquisition challenges resulted in significant test events being deferred to
later stages of the programs and later contracts
• System complexity• Ensure verification events lead to customer acceptance• Mitigate stove-piped verification planned within each sub-system test
program
• Limited test asset availability• Testing needed to be coordinated with minimal damage to system assets• Test facility also used for development – development priorities not
aligned with test team capability needs
NDIA DT&E Committee2014
18
RECOMMENDATIONS
NDIA DT&E Committee2014
19
Recommendations for mechanization
• Industry:• Understand the benefits of an effectively structure T&E program. • Include test and evaluation in the corporate management
structure.• Realize the value of having an Industry Test Lead and their
contributions to the company's overall mission and goals.• Establish an organizational culture that provides a career path for
Test and Evaluation professionals• Define Roles & Responsibilities between
• Industry Test Lead / Chief Engineer / Test IPT Lead• Continued collaboration w/ DASD(DT&E) regarding process and
infrastructure to support effective test strategies.• Government:
• Identify Industry Test Lead in RFP (Key Personnel Clause)• Structure RFP to include specific interactions between Chief
Developmental Tester and Industry Test Lead
NDIA DT&E Committee2014
20
Next Steps – Future Work
• Revisit and revise 2008 NDIA DT&E committee conducted study titled “DT&E Support to Acquisition.”
• Since the study was conducted in 2008, there has been updates to policy and new legislation (e.g. WSARA, Integrated Testing definition, updates to DOD Instruction 5000.02, etc.) which implemented some of the findings.
• Review findings and identify new/remaining gaps and/or areas for improvement.