NDIA SE Division MeetingFebruary 12, 2014
1
Developmental Test and EvaluationCommittee
Status and PlansFebruary 2014
Beth Wilson, RaytheonSteve Scukanec, Northrop Grumman
Industry Co-Chairs
NDIA SE Division MeetingFebruary 12, 2014
2
DT&E Committee2014 Status and Plans
Activity Plans for 2014 StatusT&E
Cyber Testing Industry recommendations for cyber testing
Draft completeSubmit to DASD DT&E Feb
Chief Developmental Tester
Industry interaction with new role White paper outline completeSoliciting ase study examples
Reliability Recommendations for better reliability testing
IEEE standards liaisons established
T&E Workshop Summer workshop Tutorials and papers
SE
Modeling & Simulation T&E Perspective for Modeling/ Simulation products
Joint meeting TBD
Architecture T&E Perspective for Architecture views
Joint meeting TBD
Education & Training Chief Developmental Tester role (after CDT project complete)
Systems Security Engineering
Cyber testing guidelines connections to PPP
(after SSE PPP May workshop)
SE Conference October conference DT&E track, joint tracks
Complete In Process Proposed
NDIA SE Division MeetingFebruary 12, 2014
3
Cyber Testing GuidelinesActivity Plans for 2013 - 2014 Status/Plans
Cyber Testing Review what industry does for cyber testing, expand focus to include interoperability
Recommendations for Cyber Testing guidelines
Project Lead: Dave DesjardinsJoint Project with ITEA
NDIA SE Division MeetingFebruary 12, 2014
4
Chief Developmental TesterActivity Plans for 2013-2014 Status/Plans
Chief Developmental Tester
Review service policies for new role and industry implementations
Propose model for industry interactionWhite paper T&E/SE conference presentations
The Secretary of Defense shall require that each major defense acquisition program be supported by—‘‘(A) a chief developmental tester; and‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program.
Coordinating DT&E activitiesInsight into Contractor activitiesOversee T&E activitiesInform government PM about contractor DT&E results
Project Lead: Joe Manas
NDIA SE Division MeetingFebruary 12, 2014
5
Reliability Testing
Reliability Testing Focus- Software reliability standard- Cyber testing initiatives- Data for reliability growth measures
Input/Feedback Opportunities: - IEEE meetings January and August- Periodic revision periods
Activity Plans for 2013 Status/PlansReliability Testing(with IEEE)
Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)
Solicit feedback on existing standardsIdentify recommendations to improve reliability testing
Project Leads: Lou Gullo, Taz Daughtrey
NDIA SE Division MeetingFebruary 12, 2014
6
DT&E Committee - 2014 Task PlanProjects Working Group
Proposed 2014 Tasks:• Cyber Testing Project• Chief Developmental Tester Project• Collaboration with IEEE Reliability• Joint meetings with SED Committees
• Modeling and Simulation• Architecture• Systems Security Engineering• Education and Training
Deliverables/Products• Cyber Testing Recommendations white paper• Chief Developmental Tester white paper
• Candidate KSABs for role (E&T)• Industry interaction recommendation• Tutorial on role for SE conference
• Inputs to reliability standards effort• Joint meeting notes
Schedule / Resources• Cyber Testing Draft Recommendations Feb 2014
• Weekly teleconferences started 12/3/2013• Full white paper later in 2014
• Chief Developmental Tester• Joint meeting with E&T with SED meeting• Tutorial at SE conference
• Reliability standards input for Jan/Aug meetings• Test as a Stakeholder joint meetings with SED
• Modeling and Simulation• Architecture• Systems Security Engineering
Issues / Concerns:• Industry engagement• NDIA T&E conference may be cancelled (Seattle
in March)• ITEA conference (Denver in September) may be
postponed and moved
NDIA SE Division MeetingFebruary 12, 2014
7
Backup
DT&E Committee Results 2008-2013
NDIA SE Division MeetingFebruary 12, 2014
8
T&E Themes
ForConferences
SEDCommittee
CollaborativeProjects
DT&E Committee Current Structure Since 2010
NDIA Systems Engineering Conference (Q4) Collaborative Projects
NDIA Test and Evaluation Annual Conference (Q1)Periodic T&E Committee-Focused Conferences
Cooperation with other NDIA Divisions
NDIA Industrial Committee on Test and Evaluation
(ICOTE)
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
T&E Themes & Issues
Project ResultsDT&E Themes
NDIATest and Evaluation
Division
NDIASystems Engineering
Division
NDIADT&E Committee
SE Themes & Issues
NDIA SE Division MeetingFebruary 12, 2014
9
Summary of DT&E Committee Test and Evaluation Efforts
Topic Activity
DT&E Committee
DoD T&E Policy Study 2006 – 2008: Workshops and Study ReportImproving T&E in the DoD Acquisition Process
Integrated Testing 2008 – 2010: Integrated Test Study NDIA presentations and tutorialITEA journal article
RFP Language for T&E 2010 – 2011: Comments for Update to OSD GuideIncorporating T&E into DoD Acquisition Contracts
Test Optimization 2012: Statistical Optimization Conference Thread2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing Started 2013: Industry recommendations for cyber testing
Chief Developmental Tester Started 2013: Industry interaction with new role
NDIA SE Division MeetingFebruary 12, 2014
10
Summary of DT&E Committee Systems Engineering Efforts
SED Committee Activity
SE Collaboration
Software 2009: SW T&E Summit Recommendations
System of Systems 2010 – 2011: T&E for SoS Workshop and Initiatives2012: Final Report on Best Practices Model
Performance Measurement
2012: Leading Indicators for T&E workshop
2013: Requirements Verification leading indicators presentation
Modeling and Simulation
2011: Effective Use of Modeling and Simulation for T&E 2012: Distributed Model-Based Testing
Reliability Started 2013: Recommendations for better reliability testing
NDIA SE Division MeetingFebruary 12, 2014
11
DoD T&E Policy Study August 2006: DT&E Committee Kickoff
Policy Study:“Improving T&E in the DoD Acquisition Process”Industry T&E policy recommendations
Workshops:August 2007January 2008
Focus Areas:1. Earlier contractor and tester involvement2. Integrated DT/OT and DT operational relevance3. Suitability
April 2008: Report Summarized Results:10 Findings15 Recommendations
2006-2008
NDIA SE Division MeetingFebruary 12, 2014
12
Integrated Testing (CT/DT/OT)Implementation Framework
2008-2010
NDIA SE Division MeetingFebruary 12, 2014
13
Software Test and EvaluationSoftware Summit
RFP Language
How Much Testing
is Enough?
Lifecycle and
End-to-End Software T&E
Changing Paradigms
Training & Competency ModelPolicy, Guidance, and Standards
Tools, Automation, Methodologies, Process2009
NDIA SE Division MeetingFebruary 12, 2014
14
RFP LanguageIndustry
#
Master #
Reviewer Line# Comment and Rationale Recommended Input A/R/P
Industry Comments for Update:Incorporating Test and Evaluation
Into Department of DefenseAcquisition Contracts
Recommendations from SW Summit
2010-2011
NDIA SE Division MeetingFebruary 12, 2014
15
Test and Evaluation forSystems of Systems
2010: Workshop2009: “Sleepless Nights” List of Issues
2010: “Sominex” Resulting Initiatives
InitiateSoS
PlanSoS
Update
EvolveSoSArch
EvolveSoSArch
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ConductSoS Analysis
ContinueSoS Analysis
ImplementSoS
Update
DevelopSoSArch
External Environment
2011: Best PracticesWave Model
2009-2012 2012: Final Report
NDIA SE Division MeetingFebruary 12, 2014
16
Effective Use of Modeling and Simulation for Test and Evaluation
Joint Meeting in August 2011• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data Management Working Group
• LVC Architecture Roadmap Implementation (LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group • Raytheon Presentation on M&S for T&E • Potential Topics for November AMSWG
Meeting
2011
NDIA SE Division MeetingFebruary 12, 2014
17
Modeling & Simulation forDistributed Testing
Benefits:• Find integration issues earlier• Test to learn in ‘safe’ environment• Protect proprietary information• Facilitate DT to OT transition• Increase performance testing range in operating environments • Support end to end studies throughout the programBarriers:• Security• Lack of persistent network• Early consideration of technical issues• Perceived value• Disconnect between the communities (M&S and T&E)Recommendations:• Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)• Create a framework for reusing and repurposing M&S through the product model• Establish M&S as part of statistical test design
• Determine what tests are conducted to acquire data for model validation.• Fewer test events with better models.
• Recommend the use of M&S to do I&T • Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint TrackSE Conference October 2012
Joint MeetingAugust 2012
2012
NDIA SE Division MeetingFebruary 12, 2014
18
Statistical Test Optimization2012 SE Conference summit/workshop thread- Tutorials on Monday 10/22- Presentations on Wednesday 10/24- Synthesis Panel on Wednesday 10/24
Follow-on Results: - DT&E Committee white paper - ITEA Journal article- SE Conference track Oct 2013- MORS Conference Nov 2013
2012 - 2013
NDIA SE Division MeetingFebruary 12, 2014
19
Requirements VerificationLeading Indicator Metrics
2012 SE Conference workshop- Follow-on workshop with System Performance Measurement WG - Focused on Requirements Verification not addressed in first report
Potential Measures Identified: - System Maturity Level- Verification Requirement Maturity- Technical Measures and Stakeholder Need
2012 - 2013
Recommended