Developmental Test and Evaluation Committee Status and Plans February 2014

NDIA SE Division MeetingFebruary 12, 2014

1

Developmental Test and EvaluationCommittee

Status and PlansFebruary 2014

Beth Wilson, RaytheonSteve Scukanec, Northrop Grumman

Industry Co-Chairs


2

DT&E Committee2014 Status and Plans

Activity Plans for 2014 StatusT&E

Cyber Testing Industry recommendations for cyber testing

Draft completeSubmit to DASD DT&E Feb

Chief Developmental Tester

Industry interaction with new role White paper outline completeSoliciting ase study examples

Reliability Recommendations for better reliability testing

IEEE standards liaisons established

T&E Workshop Summer workshop Tutorials and papers

SE

Modeling & Simulation T&E Perspective for Modeling/ Simulation products

Joint meeting TBD

Architecture T&E Perspective for Architecture views

Joint meeting TBD

Education & Training Chief Developmental Tester role (after CDT project complete)

Systems Security Engineering

Cyber testing guidelines connections to PPP

(after SSE PPP May workshop)

SE Conference October conference DT&E track, joint tracks

Complete In Process Proposed


3

Cyber Testing GuidelinesActivity Plans for 2013 - 2014 Status/Plans

Cyber Testing Review what industry does for cyber testing, expand focus to include interoperability

Recommendations for Cyber Testing guidelines

Project Lead: Dave DesjardinsJoint Project with ITEA


4

Chief Developmental TesterActivity Plans for 2013-2014 Status/Plans

Chief Developmental Tester

Review service policies for new role and industry implementations

Propose model for industry interactionWhite paper T&E/SE conference presentations

The Secretary of Defense shall require that each major defense acquisition program be supported by—‘‘(A) a chief developmental tester; and‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program.

Coordinating DT&E activitiesInsight into Contractor activitiesOversee T&E activitiesInform government PM about contractor DT&E results

Project Lead: Joe Manas


5

Reliability Testing

Reliability Testing Focus- Software reliability standard- Cyber testing initiatives- Data for reliability growth measures

Input/Feedback Opportunities: - IEEE meetings January and August- Periodic revision periods

Activity Plans for 2013 Status/PlansReliability Testing(with IEEE)

Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)

Solicit feedback on existing standardsIdentify recommendations to improve reliability testing

Project Leads: Lou Gullo, Taz Daughtrey


6

DT&E Committee - 2014 Task PlanProjects Working Group

Proposed 2014 Tasks:• Cyber Testing Project• Chief Developmental Tester Project• Collaboration with IEEE Reliability• Joint meetings with SED Committees

• Modeling and Simulation• Architecture• Systems Security Engineering• Education and Training

Deliverables/Products• Cyber Testing Recommendations white paper• Chief Developmental Tester white paper

• Candidate KSABs for role (E&T)• Industry interaction recommendation• Tutorial on role for SE conference

• Inputs to reliability standards effort• Joint meeting notes

Schedule / Resources• Cyber Testing Draft Recommendations Feb 2014

• Weekly teleconferences started 12/3/2013• Full white paper later in 2014

• Chief Developmental Tester• Joint meeting with E&T with SED meeting• Tutorial at SE conference

• Reliability standards input for Jan/Aug meetings• Test as a Stakeholder joint meetings with SED

• Modeling and Simulation• Architecture• Systems Security Engineering

Issues / Concerns:• Industry engagement• NDIA T&E conference may be cancelled (Seattle

in March)• ITEA conference (Denver in September) may be

postponed and moved


7

Backup

DT&E Committee Results 2008-2013


8

T&E Themes

ForConferences

SEDCommittee

CollaborativeProjects

DT&E Committee Current Structure Since 2010

NDIA Systems Engineering Conference (Q4) Collaborative Projects

NDIA Test and Evaluation Annual Conference (Q1)Periodic T&E Committee-Focused Conferences

Cooperation with other NDIA Divisions

NDIA Industrial Committee on Test and Evaluation

(ICOTE)

DT&E Committee Focus: T&E initiatives aligned with SE, DT&E

T&E Themes & Issues

Project ResultsDT&E Themes

NDIATest and Evaluation

Division

NDIASystems Engineering

Division

NDIADT&E Committee

SE Themes & Issues


9

Summary of DT&E Committee Test and Evaluation Efforts

Topic Activity

DT&E Committee

DoD T&E Policy Study 2006 – 2008: Workshops and Study ReportImproving T&E in the DoD Acquisition Process

Integrated Testing 2008 – 2010: Integrated Test Study NDIA presentations and tutorialITEA journal article

RFP Language for T&E 2010 – 2011: Comments for Update to OSD GuideIncorporating T&E into DoD Acquisition Contracts

Test Optimization 2012: Statistical Optimization Conference Thread2013: Scientific Test and Analysis Techniques report/articles

Cyber Testing Started 2013: Industry recommendations for cyber testing

Chief Developmental Tester Started 2013: Industry interaction with new role


10

Summary of DT&E Committee Systems Engineering Efforts

SED Committee Activity

SE Collaboration

Software 2009: SW T&E Summit Recommendations

System of Systems 2010 – 2011: T&E for SoS Workshop and Initiatives2012: Final Report on Best Practices Model

Performance Measurement

2012: Leading Indicators for T&E workshop

2013: Requirements Verification leading indicators presentation

Modeling and Simulation

2011: Effective Use of Modeling and Simulation for T&E 2012: Distributed Model-Based Testing

Reliability Started 2013: Recommendations for better reliability testing


11

DoD T&E Policy Study August 2006: DT&E Committee Kickoff

Policy Study:“Improving T&E in the DoD Acquisition Process”Industry T&E policy recommendations

Workshops:August 2007January 2008

Focus Areas:1. Earlier contractor and tester involvement2. Integrated DT/OT and DT operational relevance3. Suitability

April 2008: Report Summarized Results:10 Findings15 Recommendations

2006-2008


12

Integrated Testing (CT/DT/OT)Implementation Framework

2008-2010


13

Software Test and EvaluationSoftware Summit

RFP Language

How Much Testing

is Enough?

Lifecycle and

End-to-End Software T&E

Changing Paradigms

Training & Competency ModelPolicy, Guidance, and Standards

Tools, Automation, Methodologies, Process2009


14

RFP LanguageIndustry

#

Master #

Reviewer Line# Comment and Rationale Recommended Input A/R/P

Industry Comments for Update:Incorporating Test and Evaluation

Into Department of DefenseAcquisition Contracts

Recommendations from SW Summit

2010-2011


15

Test and Evaluation forSystems of Systems

2010: Workshop2009: “Sleepless Nights” List of Issues

2010: “Sominex” Resulting Initiatives

InitiateSoS

PlanSoS

Update

EvolveSoSArch

EvolveSoSArch

ImplementSoS

Update

PlanSoS

Update

ContinueSoS Analysis

ImplementSoS

Update

PlanSoS

Update


ConductSoS Analysis


ImplementSoS

Update

DevelopSoSArch

External Environment

2011: Best PracticesWave Model

2009-2012 2012: Final Report


16

Effective Use of Modeling and Simulation for Test and Evaluation

Joint Meeting in August 2011• Distributed Testing, the Joint Mission

Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA)

• DoD M&S Community of Interest Data Management Working Group

• LVC Architecture Roadmap Implementation (LVCAR‐I) Gateways Effort Applicability to T&E

• OSD T&E Working Group • Raytheon Presentation on M&S for T&E • Potential Topics for November AMSWG

Meeting

2011


17

Modeling & Simulation forDistributed Testing

Benefits:• Find integration issues earlier• Test to learn in ‘safe’ environment• Protect proprietary information• Facilitate DT to OT transition• Increase performance testing range in operating environments • Support end to end studies throughout the programBarriers:• Security• Lack of persistent network• Early consideration of technical issues• Perceived value• Disconnect between the communities (M&S and T&E)Recommendations:• Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)• Create a framework for reusing and repurposing M&S through the product model• Establish M&S as part of statistical test design

• Determine what tests are conducted to acquire data for model validation.• Fewer test events with better models.

• Recommend the use of M&S to do I&T • Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T

Joint TrackSE Conference October 2012

Joint MeetingAugust 2012

2012


18

Statistical Test Optimization2012 SE Conference summit/workshop thread- Tutorials on Monday 10/22- Presentations on Wednesday 10/24- Synthesis Panel on Wednesday 10/24

Follow-on Results: - DT&E Committee white paper - ITEA Journal article- SE Conference track Oct 2013- MORS Conference Nov 2013

2012 - 2013


19

Requirements VerificationLeading Indicator Metrics

2012 SE Conference workshop- Follow-on workshop with System Performance Measurement WG - Focused on Requirements Verification not addressed in first report

Potential Measures Identified: - System Maturity Level- Verification Requirement Maturity- Technical Measures and Stakeholder Need

2012 - 2013

Documents

Developmental Test and Evaluation Committee Status and Plans February 2014