Upload
ferdinand-byrd
View
219
Download
0
Embed Size (px)
Citation preview
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
1TOC
NDIA Systems Engineering DivisionAnnual Planning Meeting
2013 Summary and 2014 Plans
Action Plan StatusCommittee Reports
December 11-12, 2013
With placeholders for inputs from OSD and Services - TBD
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
2TOC
Introduction
Status of 2013 NDIA SE Division activities• Relative to 2013 Plan• SED plan from 12/12 meeting,
briefed at 2/13 meeting
Committee Reports• 2013 Accomplishments• 2014 Plans
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
3TOC
NDIA SED 2013 Tasks- Derived from DoD/Services Initiatives (Voice of the Customer)
ASA (ALT)OCSE
• Dev Planning• SE Workforce• SE methods/
tools• Reliability• Security
protection• ExportabiIity• Soldier/Small
Unit Challenge
DASN RDT&E SE&P 2012
YSTEMSNGINEERING
OLICY
Questions NDIA can help answer
1. Identify the new skills required of an engineer to succeed on an Agile development team
2. Recognizing that we can’t always create the ideal team, what processes can be developed to assist engineers on an Agile development team
3. The System of Systems mission level requirements analysis process has been identifying requirements that satisfy a mission and allocating those requirements to systems. The relationship between those allocated requirements and the application of new R&D products to meet those requirements is not robust.
4. How can the Navy work with Industry to improve the assessment of the compendium of R&D products* for application to system requirements, to increase success of the Technology Demonstration acquisition phase?
5
DASN (RDT&E)SEP
• Agile in acquisition
• SoS mission reqts
• R&D assessment
DoD Sys Eng2012/12/12 | Page-12
Distribution Statement A – Approved for public release by OSR, SR Case # 12-S-3121 applies.
Summary of DASD(SE) Goals
• Continued excellence in SE support to programs and acquisition decisions
• Improved, consistent PPP engagement with programs resulting in successful protection strategies
– New processes for Software Assurance, Trusted Microelectronics, Exportability
• Advocacy and impact in ensuring SE workforce capacity and capability
• Provide depth to acquisition policy and processes with SE guidance, practices, and continuous learning opportunities
• Make progress on areas of DoD interest:– Quantitative Risk Management
– SE Tradespace Analysis
– Supply Chain Risk Management and Damage Assessment
– Anti-Counterfeit
– Advanced Modeling for Engineering Resiliency
DASD (SE)• ERS• Anti-
Counterfeit• Risk Mgmt• Schedule risk• Supplier risks• Security
design• SE models
(MBx)• SoSE
VerbalInputs
Provided
SAF AQRE• Policy (levels)• Process• Tradespace• Architecture• Cost
Assessment• Specialty Engrg• People /
Workforce (specialists vs. generalists)
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
4TOC
NDIA SED 2013 Action Plan Status
# NDIA SED 2013 Tasks Committee Plan
1 Quantitative risk management guidance. [DASD(SE)]
SEEC(Brown, Rassa)
Ongoing action. Feb workshop planned.Risk workshop planned, deferred.No action in 2012 or 2013 – drop it?
2 Support Engineered Resilient Systems (ERS) initiatives. Link ERS with M&S? [DASD(SE)]
SED (Rassa)M&S
IEEE/NDIA ERS workshop in planning. Other actions pending. Division organizational emphasis?No action yet, still pending.
3 Anti-counterfeit – SE role, industry feedback for supply chain risk management, augment body of knowledge [DASD(SE)]
SED(Henry)
Issue SED request for 1:1 company inputs provided direct to DASD(SE). Potential follow-up workshop. Could be company proprietary? Coordinate with other NDIA, AIA, etc.No request from OSD; no action
4 Design Patterns for Security – share SERC research findings (UVA)[DASD(SE)]
SED(Henry)
Schedule Dr. Horowitz (UVA) for NDIA SED briefing. Done, Feb 2013 SED meeting.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
5TOC
NDIA SED 2013 Action Plan Status
# NDIA SED 2013 Tasks Committee Plan
5 Advanced Modeling: Model Based for “X” (MBx) – how could acquisition strategies be enhanced to use models (RFPs, etc.) [DASD(SE)]
M&S(Coolahan)
M&S committee study or report. Collect industry input, examples.See M&S plan.
6 SoSE: how to demand articulation and insertion of SoS context during life cycle acquisition? [DASD(SE)]
SoS (Dahmann)
SoS committee study or report and recommendations. International issues?
7 Soldier / Small Unit Challenge[ASA(ALT)]
SED WG(Serna)
WG to consider soldier and small unit architecture.Discontinued; lack of Army support
8 Design for Reliability – need RFP language. [ASA(ALT)]
LCS(Cothran)
Consider input on RFP language.LCS committee is inactive; no chair
9 Security / Protection for coalition operations [ASA(ALT)]
System Assurance (Croll)
SA committee to consider security issues for coalition – provide recommendations to SED for actionSA committee rechartered to SSE
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
6TOC
NDIA SED 2013 Action Plan Status# NDIA SED 2013 Tasks Committee Plan
10 Design for Exportability – BBP 2.0, foreign military sales (FMS), direct commercial sales (DCS) [ASA(ALT)]
No SED action
11 Reconcile Agile development with DoD 5000.2 acquisition practices, requirements, IPT structure, cost, skills[DASN (RDT&E)]
SW(Draper)
Conduct workshop. Collect best practices (cross-discipline, SEI, DHS, AFEI, INCOSE, …Partner with SEI, ADAPT. See Agile guidance. Kickoff ‘Agile and SE’ at NDIA SE Conf – WG formation in 2014.
12 Consider how to move tradespace for architecture early in the life cycle (pre-MDD)
Architecture (Sheeley)MA (Lohse)
Start initially with white paper vision on use of architecture to bound/expand the trade space (solution agnostic)See Arch and MA status briefings
13 Collect industry best practices for growing and retaining specialists vs. generalists
E&T (Gelosh) How does industry keep its specialists?See E&T briefing for status
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
7TOC
NDIA SED 2014 Actions
# NDIA SED 2013 Tasks Committee Plan / Status
1 Brief the existing M&S life cycle cost model report at the NDIA SED February meeting
M&S(Bergenthal, Coolahan)
Plan for Feb 2014 NDIA SED meeting.
2 Review the Top 5 SE issues report on technical authority and consider if any NDIA actions are appropriate. Consider best practices (industry or government) to implement technical authority.
NDIA SED(Henry, Rassa, Draper, Serna)
3 Request DoD briefing on how program startups work (Dave Ahern). Provide industry best practices on program startup/assist teams.
NDIA SED(Henry, Rassa)
Schedule for NDIA SED meeting briefings.
4 Support MORS affordability handbook development and community of practice initiatives.
Affordability WG / NDIA SED
Serna
5 NDIA to provide recommendations on deployment of SE 15288.1 and 15288.2 standards (e.g., RFP language, transition, governance, compliance).
NDIA SED(Draper / Rassa / Roedler / Henry)
Coordinate with joint IEEE/DoD/industry WG.
6 Assess new DoDI 5000.02 and provide recommendations on any NDIA implementation actions needed (impact analysis).
SEEC (Brown) or short term WG
Focus on industry clarification, adoption and impact analysis.
7 Collect industry best practices for quantitative risk management.
NDIA SED(Rassa, Forbes)
Establish risk management WG, with meetings held following NDIA SED meetings.
8 Support Engineered Resilient Systems (ERS) initiatives.
NDIA SED / IEEE(Rassa)
Follow on from open 2013 actions. IEEE activity with NDIA support.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
8TOC
NDIA SED 2014 Actions
# NDIA SED 2013 Tasks Committee Plan / Status
9 Schedule committees for special topics and detailed briefs at NDIA SED meetings.
NDIA SED (Henry) Feb: SoS – Aug?April: SSE (PPP) – June?June: M&SAugust: E&TOctober (SE Conf plenary?): DT&E
10 Consider ways to increase NDIA SED collaboration internally (committees, other NDIA divisions) as well as externally (other organizations, company representatives)
NDIA SED Consider a consolidated and managed list of NDIA external stakeholders at the SED level. Ensure integration without duplicating effort.
11 Several committees could use more industry and government participation – consider ways to build more involvement, given current constraints.
NDIA SED Collect all needs and descriptions in a consolidated request to Eng VPs?
12 Document and publicize the procedures for releasing reports through NDIA. Identify which committee products are in draft form that need to be formalized.
NDIA SED (Rassa, Britt)
Reports need to go through NDIA (Britt) prior to sending to the SED steering group for review. For web publishing send to Britt Bommelje and Rebecca Danahy.
13 Britt to provide a copy of the NDIA SED charter. NDIA Distributed via email.
14 Look at M&S use assessment and identify areas for further tasks to support MBSE.
M&S This could be useful to drive other research.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
9TOC
NDIA SED 2014 Actions
# NDIA SED 2013 Tasks Committee Plan / Status
15 Some NDIA SED web pages are badly out of date – SED leadership and committee to update web pages under their areas of control.
NDIA SED / Committee Leads
16 ODASD(SE) to provide guidance on next steps for SE Effectiveness study (e.g., briefing SAEs)
NDIA SED / ODASD(SE)
Dona to provide feedback.
17 Consider a follow-on to the root cause analysis study of PSR findings.
NDIA SED / ODASD(SE)
Potential POCs: Bob Rassa, Jim Thompson, Pete Nolte, Gary Bliss, …
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
10TOC
Committee Inputs- 2013 Task Progress / 2014 Task Plans
Architecture (Arch)Automatic Test (ATC)Development Test and Evaluation (DT&E)Education and Training (E&T)Enterprise Health Mgmt (EHM)Environment, Safety, and Occupational Health (ESOH)Human-System Integration (HSI) – no slidesJoint Committee on Systems Engrg in Manufacturing (JCSEM) – no slidesLife Cycle Support (LCS) – no slidesModeling and Simulation (M&S)Mission Analysis (MA)Systems Engineering Effectiveness Committee (SEEC)Software CommitteeSystem of Systems (SoS)Systems Security Engineering (SSE)
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
11TOC
Architecture Committee – 2013 Task Summary
• Meetings:• 7 Virtual meetings• 1 Face-to-face meeting
• SE Conference:• 15 papers
• Accomplishments:• Completed the DoDAF Recommendations Report
• Delivered it to Steve Henry• Posted on the website
• Presented Architecture Metrics results (worked in 2012) at the INCOSE Symposium• Participated in SEI SoS Architecture workshop• Received architecture informational briefings / demos from C.Gedo, Architecture
Stnds Group (DISA & GTRI), and on J6 Architecture portal from B.Piazza (JS J6)
• In-work:• Reviewing whitepaper “Role of Architecture to Influence the Development Planning
Tradespace” with the Mission Analysis subcommittee• Open system architecture discussions with Dr. Flietz (AFRL)
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
12TOC
Architecture Committee - 2014 Task Plan
Proposed 2014 Tasks:
• Complete joint Architecture/Mission Analysis white paper “Role of Architecture to Influence the Development Planning Tradespace”
• Investigate status and make proposals around subject of Open System Architectures
• Initiate joint discussions with Test Committee
Deliverables/Products
• “Role of Architecture to Influence the Development Planning Tradespace” whitepaper
Schedule / Resources
• Release whitepaper 1Q• Initiate Open System Architecture discussions
1Q• Initiate joint Architecture – Test discussions 1Q
Issues / Concerns:
• Availability of committee members
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
13TOC
Automatic Test Committee – 2013 Status
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
14TOC
Automatic Test Committee - 2014 Task PlanProjects Working Group
Proposed 2014 Tasks:• Government/Industry Liaison Reports
• Detail, Staff, Schedule & Initiate new ATC tasks:• Industry Survey for Smart Automatic Test
System Opportunities• Software Architecture for Synthetic
Instrumentation
Deliverables/Products
• Semi-annual Reports posted on ATC Web-site:• US Army ATS Liaison Report• USAF ATS Liaison Report• US Navy/USMC-Air ATS Liaison Report• USMC-Ground ATS Liaison Report• DoD AMB Liaison Report• Commercial ATS Liaison Report
• Deliverables from new tasks: TBD
Schedule / Resources
Status/Review/Present at regular ATC meetings:• March/April (adjacent DoD ATS Management
Board (AMB) Meeting – Gov’t participating committee) at NDIA HQ
• September in St Louis (AUTOTESTCON)• December with SED (TBD)
No resource issues or support anticipated
Issues / Concerns:• Only one concern specific to ATC:
• Reduced defense budgets and fallout from the Las Vegas GSA meeting has had a significant impact on Government (AMB) participation in many events involving travel – not just NDIA - including holding virtual AMB meetings.
• Response:• ATC is holding its meeting at NDIA HQ
adjacent AMB scheduled meetings, and will remain flexible in scheduling our meetings to maximize attendance, participation, etc.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
15TOC
Developmental Test and EvaluationCommittee
Status and Plans December 2013
Beth Wilson, RaytheonSteve Scukanec, Northrop Grumman
Industry Co-Chairs
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
16TOC
DT&E Committee2013 Status
Activity Plans for 2013 Status/Plans
Scientific Test and Analysis TechniquesSTAT for T&E
Effective use of statistical approaches for test optimization (including DOE) with examples
White paper completePreparing journal publicationsPresented results at SE conference
Modeling & Simulation
Evaluate test as a stakeholder Joint meetings with SED meetingsPropose for 2014
Architecture
NDIA T&E Conf DT&E panel, STAT report Conference cancelled
NDIA SE Conf DT&E tracks Joint tracks, metrics/STAT reports
Added Task Dec 2012: Metrics
Provide more information on leading indicators for T&E
Status: draft reportPlans: results at SE conference
Added Task Dec 2012: Reliability
Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)
Discussed at June 2013 meetingLeads identified Nov 2013
Added Task Jun 2013:Cyber Testing
Develop recommendations for Cyber Guidelines 2.0 in 2014
Lead and team identifiedKicked off Nov 2013 (joint with ITEA)
Added Task Jun 2013:Chief Developmental Tester
Develop recommendations for industry interaction with Chief Developmental Tester role
Lead and team identified
Complete In Process Cancelled
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
17TOC
DT&E Committee - 2014 Task PlanProjects Working Group
Proposed 2014 Tasks:
• Cyber Testing Project• Chief Developmental Tester Project• Collaboration with IEEE Reliability• Joint meetings with SED Committees
• Modeling and Simulation• Architecture• Systems Security Engineering• Education and Training
Deliverables/Products
• Cyber Testing Recommendations white paper• Chief Developmental Tester white paper
• Candidate KSABs for role (E&T)• Industry interaction recommendation• Tutorial on role for SE conference
• Inputs to reliability standards effort• Joint meeting notes
Schedule / Resources
• Cyber Testing Draft Recommendations Feb 2014• Weekly teleconferences started 12/3/2013• Full white paper later in 2014
• Chief Developmental Tester• Joint meeting with E&T with SED meeting• Tutorial at SE conference
• Reliability standards input for Jan/Aug meetings• Test as a Stakeholder joint meetings with SED
• Modeling and Simulation• Architecture• Systems Security Engineering
Issues / Concerns:
• Industry engagement• NDIA T&E conference may be cancelled (Seattle
in March)• ITEA conference (Denver in September) may be
postponed and moved
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
18TOC
Summary of DT&E Committee2013 Results/2014 Plans
Topic Activity
T&E
Test Optimization 2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing 2013-2014: Industry recommendations for cyber testing
Chief Developmental Tester 2013-2014: Industry interaction with new role
SE
Performance Measurement 2013: Requirements Verification leading indicators report
Reliability 2013-2014: Recommendations for better reliability testing
Modeling and Simulation 2014: T&E Perspective for Modeling/Simulation products
Architecture 2014: T&E Perspective for Architecture views
Education and Training 2014: Chief Developmental Tester role
Systems Security Engineering
2014: Cyber testing guidelines connections to Program Protection Planning
Completed Current Proposed
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
19TOC
DT&E Committee 2012-2013Statistical Test Optimization
2012 SE Conference summit/workshop thread- Tutorials on Monday 10/22- Presentations on Wednesday 10/24- Synthesis Panel on Wednesday 10/24
Follow-on Results: - DT&E Committee white paper - ITEA Journal article- SE Conference track Oct 2013
Activity Plans for 2013 Status/Plans
Scientific Test and Analysis TechniquesSTAT for T&E
Examples of effective use of statistical approaches for test optimization (including DOE) for an implementation framework
White paper completeSE Conference track: 2012 and 2013ITEA journal publication MORS conference Nov 2013
2012 - 2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
20TOC
DT&E Committee 2012-2013Leading Indicator Metrics
2012 SE Conference workshop- Follow-on workshop with System Performance Measurement WG - Focused on Requirements Verification not addressed in first report
Potential Measures Identified: - System Maturity Level- Verification Requirement Maturity- Technical Measures and Stakeholder Need
Activity Plans for 2013 Status/Plans
Metrics(with Performance Measurement WG)
Provide more information on leading indicators for T&E
Draft reportPresented results at SE conference
2012 - 2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
21TOC
DT&E Committee 2012-2013Reliability Testing
Reliability Testing Focus- Software reliability standard- Cyber testing initiatives- Data for reliability growth measures
Input/Feedback Opportunities: - IEEE meetings January and August- Periodic revision periods
Activity Plans for 2013 Status/Plans
Reliability Testing(with IEEE)
Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)
Solicit feedback on existing standardsIdentify recommendations to improve reliability testing
Project Leads: Lou Gullo, Taz Daughtrey
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
22TOC
DT&E Committee 2013-2014Cyber Testing
Activity Plans for 2013 - 2014 Status/Plans
Cyber Testing Review what industry does for cyber testing, expand focus to include interoperability
Recommendations for Cyber Testing guidelines
Project Lead: Dave DesjardinsJoint Project with ITEA
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
23TOC
DT&E Committee 2013-2014Chief Developmental Tester
Activity Plans for 2013-2014 Status/Plans
Chief Developmental Tester
Review service policies for new role and industry implementations
Propose model for industry interactionWhite paperT&E/SE conference presentations
The Secretary of Defense shall require that each major defense acquisition program be supported by—‘‘(A) a chief developmental tester; and‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program.
Coordinating DT&E activitiesInsight into Contractor activitiesOversee T&E activitiesInform government PM about contractor DT&E results
Project Lead: Joe Manas
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
24TOC
Backup
DT&E Committee Results 2008-2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
25TOC
Summary of DT&E Committee Test and Evaluation Efforts
Topic Activity
DT&E Committee
DoD T&E Policy Study 2006 – 2008: Workshops and Study ReportImproving T&E in the DoD Acquisition Process
Integrated Testing 2008 – 2010: Integrated Test Study NDIA presentations and tutorialITEA journal article
RFP Language for T&E 2010 – 2011: Comments for Update to OSD GuideIncorporating T&E into DoD Acquisition Contracts
Test Optimization 2012: Statistical Optimization Conference Thread2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing Started 2013: Industry recommendations for cyber testing
Chief Developmental Tester Started 2013: Industry interaction with new role
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
26TOC
Summary of DT&E Committee Systems Engineering Efforts
SED Committee Activity
SE Collaboration
Software 2009: SW T&E Summit Recommendations
System of Systems 2010 – 2011: T&E for SoS Workshop and Initiatives2012: Final Report on Best Practices Model
Performance Measurement
2012: Leading Indicators for T&E workshop
2013: Requirements Verification leading indicators report
Modeling and Simulation
2011: Effective Use of Modeling and Simulation for T&E 2012: Distributed Model-Based Testing
Reliability Started 2013: Recommendations for better reliability testing
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
27TOC
T&E Themes
ForConferences
SEDCommittee
CollaborativeProjects
DT&E Committee Current Structure Since 2010
NDIA Systems Engineering Conference (Q4) Collaborative Projects
NDIA Test and Evaluation Annual Conference (Q1)Periodic T&E Committee-Focused Conferences
Cooperation with other NDIA Divisions
NDIA Industrial Committee on Test and Evaluation
(ICOTE)
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
T&E Themes & Issues
Project ResultsDT&E Themes
NDIATest and Evaluation
Division
NDIASystems Engineering
Division
NDIADT&E Committee
SE Themes & Issues
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
28TOC
DoD T&E Policy Study
August 2006: DT&E Committee Kickoff
Policy Study:“Improving T&E in the DoD Acquisition Process”Industry T&E policy recommendations
Workshops:August 2007January 2008
Focus Areas:1. Earlier contractor and tester involvement2. Integrated DT/OT and DT operational relevance3. Suitability
April 2008: Report Summarized Results:10 Findings15 Recommendations
2006-2008
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
29TOC
Integrated Testing (CT/DT/OT)Implementation Framework
2008-2010
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
30TOC
Software Test and EvaluationSoftware Summit
RFP Language
How Much Testing
is Enough?
Lifecycle and
End-to-End Software T&E
Changing Paradigms
Training & Competency Model
Policy, Guidance, and Standards
Tools, Automation, Methodologies, Process2009
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
31TOC
RFP Language
Industry
#
Master #
ReviewerLine
#Comment and Rationale Recommended Input A/R/P
Industry Comments for Update:Incorporating Test and Evaluation
Into Department of DefenseAcquisition Contracts
Recommendations from SW Summit
2010-2011
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
32TOC
Test and Evaluation forSystems of Systems
2010: Workshop2009: “Sleepless Nights” List of Issues
2010: “Sominex” Resulting Initiatives
InitiateSoS
PlanSoS
Update
EvolveSoSArch
EvolveSoSArch
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ConductSoS Analysis
ContinueSoS Analysis
ImplementSoS
Update
DevelopSoSArch
External Environment
2011: Best PracticesWave Model
2009-2012 2012: Final Report
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
33TOC
Effective Use of Modeling and Simulation for Test and Evaluation
Joint Meeting in August 2011• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data Management Working Group
• LVC Architecture Roadmap Implementation (LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group
• Raytheon Presentation on M&S for T&E
• Potential Topics for November AMSWG Meeting
2011
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
34TOC
Modeling & Simulation forDistributed Testing
Benefits:• Find integration issues earlier• Test to learn in ‘safe’ environment• Protect proprietary information• Facilitate DT to OT transition• Increase performance testing range in operating environments • Support end to end studies throughout the programBarriers:• Security• Lack of persistent network• Early consideration of technical issues• Perceived value• Disconnect between the communities (M&S and T&E)Recommendations:• Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)• Create a framework for reusing and repurposing M&S through the product model• Establish M&S as part of statistical test design
• Determine what tests are conducted to acquire data for model validation.• Fewer test events with better models.
• Recommend the use of M&S to do I&T • Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint TrackSE Conference October 2012
Joint MeetingAugust 2012
2012
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
35TOC
Education and Training Committee – 2013 Task Summary
• Established a Joint Competency Working Group (JCWG)
• Includes members from E&T Committee and INCOSE Competency WG
• Main objective is to evolve to a globally accepted* and marketed standard competency framework, based on systems engineering effectiveness, that can be used to produce competency models tailored to the needs of the customer organizations.
• Developed white paper and proposal to evolve current INCOSE UK Competencies Framework along four paths:
• Evolve to an SE role-based competency framework that is extensible, scalable, and tailorable by the customer organization.
• Evolve by adding the concept of classes. This is where the competencies achieve their interdependence with each other.
• Evolve by including a Professional Class that covers leadership and soft skill competencies.
• Evolve by adding a new level of proficiency called the Senior Practitioner to the existing four levels in the INCOSE UK Competencies Framework.
* Approved for release as an INCOSE product.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
36TOC
Education and Training Committee - 2014 Task Plan
Proposed 2014 Tasks:• Develop crisp definitions of Knowledge, Skills,
Abilities, Behaviors and Experience.• Develop better definitions of the six classes.• Refine definition of the Senior Practitioner level.• Develop better definitions for each version (0.5,
0.75, and 1.0).• Explore how system complexity affects SE Roles. • Develop use cases.• Ensure consistency with INCOSE Certification
program and SE Handbook.
Deliverables/Products
• Competencies Framework V 0.5• Competencies Framework V 0.75• Competencies Framework V 1.0• Conference Papers and Presentations• Final Report
Schedule / Resources:
• Two weeks before INCOSE IW 2014, send draft Version 0.5 to JCWG members.
• Conduct working session at IW 2014 to finalize Version 0.5.
• Conduct working session at IS 2014 to finalize Version 0.75
• Ultimate Goal: Produce Version 1.0 by IW 2015.
Issues / Concerns:
• Few outside UK use framework – Need to seek broader buy-in early and throughout development process and show how we can use evolved competencies framework to clarify what a SE is, what they do, and why they are important.
• Project may face analysis paralysis – Need to set reasonable POA&M, stick with schedule, discuss, move forward, exhibit strong leadership along the way.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
37TOC
EHM Committee – 2013 Task Summary
2013 Tasks Planned Status Accomplishments (deliverables, etc.)- Comments
• Integrated Diagnostics Handbook update
In-work Accomplishments: OSD/ATL has been working on rewrite of Mil-Hdbk-1814. EHM participation from May 2013. Effort to continue through mid 2014
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
38TOC
(EHM Committee - 2014 Task Plan
Proposed 2014 Tasks:
• working with OSD and rewrite of Integrated Diagnostic handbook
Deliverables/Products
• Review/Comments of Integrated Diagnostics Handbook as it progresses
Schedule / Resources
Status/Review/Present at regular EHM meetings:• March/April EHM Meeting• September EHM Meeting • December with SED
No resource issues or support anticipated
Issues / Concerns:
• No Risks, Barriers or Concerns specific to EHM; However, reduced defense budgets has had a significant impact on Government, and defense contractor participation in many events
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
39TOC
ESOH Committee – 2013 Task Summary
Completed 2013 Tasks:• Supplemented the new Defense Acquisition Guidebook Chapter 4, Systems Engineering
• Teaming with DoD Acquisition ESOH IPT• “Handbook for Human Systems Integration (HSI) & Environment, Safety, and Occupational Health
(ESOH) in Pre- Milestone A Analysis of Alternatives (AoA) and Joint Capabilities Integration and Development System (JCIDS) Activities,” published October 2013
• “Guide to ESOH in the Systems Engineering Plan (SEP); Programmatic Environment, Safety, and Occupational Health Evaluation (PESHE); and National Environmental Policy Act (NEPA)/Executive Order (EO) 12114 Compliance Schedule,” published October 2013
• Coordinated with Aerospace Industrial Association (AIA) on rewrite of National Aerospace Standard 411 (NAS411)• Teaming with DoD Acquisition ESOH IPT • Goal of rewrite: provide detailed implementation instructions to contractors on implementation of the
MIL-STD-882E, DoD Standard Practice for System Safety, Task 108, "Hazardous Materials Management Plan"
• NAS 411, “Hazardous Materials Management Program,” published 30 September 2013• NAS 411-1, “Hazardous Materials Target List,” published 30 September 2013• Key element: List of hazardous materials categorized per Task 108 as Prohibited or Restricted
• Planned the NDIA SE Conference ESOH Track• Combined HSI and ESOH Track and coordinated abstracts with the HSI Committee• Six abstracts to address MRAP Lessons Learned for Rapid and Urgent Program ESOH
management, streamlining ESOH documentation initiatives, and updates on DoD Policy
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
40TOC
ESOH Committee - 2014 Task Plan
Proposed 2014 Tasks:• Focus on improving implementation of current
DoD Acquisition SE ESOH policy• Continue work with AIA on NAS 411-1 list of
Tracked materials• Work with DSOC/SOH Council on memo
regarding 15 Nov 2013 Army CAE memo on ESOH risk management and reporting
• MIL-STD-882E implementation work• Develop mechanism and collect Industry and
Government feedback on policy and guidance• Quarterly webinars
Deliverables / Products
• List of Tracked Hazardous Materials
• Memo co-signed by P&R and DUSD (I&E)
• MIL-STD-882E corrections and DIDs• Comments on improving DoD ESOH policy and
guidance, and MIL-STD-882E• Webinars
Schedule / Resources• NAS 411-1 list of Tracked materials – ECD Dec
2014/funding by AIA and DoD participants• Memo regarding 15 Nov 2013 Army CAE memo –
ECD Jun 2014/funding by DoD participants • MIL-STD-882E implementation work – TBD• Develop mechanism and collect Industry and
Government feedback on policy and guidance – ECD Dec 2014/funding by participants
• Quarterly webinars – ECD June 2014/DoD funded
Issues / Concerns:• Agreement between DoD and AIA on Tracked list
• Time for staffing and advocacy for proposed P&R and DUSD (I&E) memo
• Funding for MIL-STD-882E update development• Time and funding for questionnaire development
• OSD support
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
41TOC
Modeling and Simulation Committee – 2013 Task Summary
• Final report on M&S use assessment- Briefing on product given at October 2013 Systems Engineering
Conference- Key spreadsheet completed and shared with ODASD/SE/SA
liaison- Final report draft in progress – to be completed in mid-December
• Committee meetings- Meetings held in February, April, June, and August 2013- Presentations posted on committee web site
• Report on essential elements of a System Model- Subcommittee formed in August 2013- Expect final report in July 2014
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
42TOC
Modeling and Simulation Committee - 2014 Task Plan
Proposed 2014 Tasks:• Complete study to determine essential
elements of a “System Model” that evolves over the life cycle (started in 2013)
• Provide continuing situational awareness for M&S Committee members (ongoing)
• Motivate submittal of abstracts to populate full-length M&S track at 2014 SE Conference
Deliverables/Products
• Final report on essential elements of a System Model
• Committee meeting presentations posted on committee web site
Schedule / Resources
• Final report on essential elements of a System Model – July 2014• Have some volunteers; need to recruit
more• Committee meetings in February, April,
June, and August 2014• Need organizations to volunteer to host
Issues / Concerns:
• Finding volunteers to work on unfunded study efforts is getting much more difficult in current fiscal environment.
• Continuing Government restrictions on travel and fees for professional society activities is creating difficulties in maintaining perception of value of society activities to industry participants.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
43TOC
Mission Analysis Committee – 2013 Task Summary
• Collaborated with ASD R&E/Research on the integration of S&T/IR&D Industry involvement in the Multidisciplinary University Research
Initiative (MURI) process Increase Industry awareness and use of Defense Innovation
Marketplace• Developed draft whitepaper on “The Role of Architecture in the
Development Planning Trade Space” with the Architecture Committee
• Collaborated with the Government DPWG on Development Planning Activities Models Best Practices, Techniques, Methodologies, and Tools
• Completed all actions from June 2012 NDIA DPWG Workshop, except: Intellectual Property as a barrier to successful DP OCI as a barrier to successful DP Final DPWG Workshop Report not submitted
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
44TOC
Mission Analysis Committee - 2014 Task Plan
Proposed 2014 Tasks:
• Support the Government DPWG and ASD R&E/Research on an as needed basis
• Organizational Conflict of Interest (OCI) Task Collaborate with the Government DPWG to
address OCI Develop a clear and concise understanding of
the FAR / DFARS language Communicate findings across Gov’t/ Industry
• Complete the “Role of Architecture in Development Planning” Whitepaper with the Architecture Committee
Deliverables/Products
• “Role of Architecture in Development Planning” Whitepaper w/ the Architecture Committee
• Final Report on OCI Task findings and recommendations
Schedule / Resources
• “Role of Architecture in Development Planning” Whitepaper – April 2014
• OCI Findings and Recommendations – June, 2014
• OCI Final Report – August, 2014
Issues / Concerns:
• Diminishing number of contributing committee members
• Developing Planning instantiation across Government and Industry has been largely successful. It is questionable as to the need to continue the Mission Analysis Committee. Resources may be better utilized elsewhere.
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
45TOC
SE Effectiveness Committee– 2013 Task Summary
• Widely disseminated the results of the SE Effectiveness Study through numerous channels
• Released detailed survey results report• Discussed follow-on work ideas with DASD(SE)
• SE self-assessment tool for programs• “Lightweight” Pre-PSR program assessment
• Government presentationso DASD(SE)o USD(AT&L)o DoD SE Forumo Army SE Forumo DoD DPWG
• Conference presentationso IEEE Syscono INCOSE IS2013o NDIA GVSETSo NDIA SE Conference (panel)
• Presentations for INCOSE chapterso Finger Lakes chaptero Space Coast chaptero Chicagoland chapter
• Corporate presentationso Westinghouse Nuclearo Northrop Grumman Corp.o Harris Corp.o Boeing
• Other presentationso MITREo INCOSE webinaro Podcast for SEI
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
46TOC
Systems Engineering Effectiveness Committee – 2014 Task Plan
Proposed 2014 Tasks:
Task 1: Develop a means of measuring risk management process value
- Enable the evaluation of risk management in programs
- Enable the comparison of alternate risk management paradigms
- Enable the improvement of risk management processes• Task 2: “Get this* out to the SAEs.”, (awaiting further
instructions from DASD (SE) – some possibilities listed below)
- Periodic self-assessment of programs- Pre- Program Support Reviews (PSR) assessment of
programs- Task 3: Establish collaboration effort with INCOSE SE
Effectiveness Working Group
* The SE Effectiveness Study results (per The Honorary Frank Kendall, USD (AT&L))
Deliverables/Products
• Task 1: Presentation at Annual SE Conference• Task 2: Report out at year-end Planning Meeting• Task 3: TBD
Schedule / Resources
• Task 1: Oct 2014• Task 2: Dec 2014• Task 3: TBD • Task 1 & 2: Assigned liaison from DASD (SE)
office
Issues / Concerns:
• Working agreement with INCOSE SE Effectiveness Working Group
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
47TOC
SoS SE Committee 2013 Task Summary
• Committee inputs to TTCP Product addressing SoS at key points in the systems development process• February 2013: Initial presentation of TTCP draft • March 2013: Inputs from committee members on TTCP draft• November 2013: Shared Updated Product, with request for added inputs• December 2013: Inputs coming in now
• SoS Webinar Series: SoS Engineering Collaborators Information Exchange (SoSECIE)
• 14 Webinars, each with between 40-100 participants• Presentations are posted on SoS SE Website http://
www.acq.osd.mil/se/outreach/sosecollab.html
• Committee participation SEI workshop on SoS SW Reuse Patterns• Committee Meetings focused on Issues of Interest
• February - TTCP Product Review• April – T-Area-SoS – EU SoS Research Agenda Development Projects• July – Army Victory Architecture (with Architecture Committee)• August – M&S and SoS (with M&S Committee)
• Six SoS tracks at SE Conference, most joint with other committees• Drafted 15288 annex for systems of systems accepted for revision
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
48TOC
SoS Committee2014 Task Plan
Proposed 2014 Tasks
• Input and Feedback from TTCP• Regular committee exchanges
Focus on new and emerging Service SoS Initiatives
Joint meeting with SSE on SoS/SSE
• SoS Webinar Series: SoS Engineering Collaborators Information Exchange (SoSECIE)
Deliverables/Products
• Feedback on inputs to TTCP work product
• Presentations on SoS SE Committee Area on NDIA Website
• Webinars slides posted on DASD SE site
Schedule / Resources
• Bi-Monthly committee exchanges• Bi-Weekly Webinars
Issues / Concerns• None
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
49TOC
Systems Security EngineeringCommittee
Status and Plans December 2013
Holly Coulter Dunlap, RaytheonBeth Wilson, Raytheon
Industry Co-Chairs
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
50TOC
SSE Committee2013 Status: NEW
Activity Plans for 2013 Status/Plans
Added Task Apr 2013:New SSE Committee
Continue work of Systems Assurance Committee to follow up on May 2012 Program Protection Planning (PPP) workshop
Kickoff held June 18, 2013Committee renamed Systems Security Engineering (SSE) Committee
NDIA SE Conference: SSE track, joint SSE/SoS trackProgress on 5 priorities identified May 2012
Follow-on Workshop planned for May 2013
Complete In Process Cancelled
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
51TOC
SSE Committee - 2014 Task PlanProjects Working Group
Proposed 2014 Tasks:
• PPP Implementation Workshop• Joint meetings with SED Committees
• Developmental Test and Evaluation: Connections between PPP and cyber testing guidelines
• Systems of Systems: PPP leverage points in the SoS Wave Model
Deliverables/Products
• Workshop Recommendations • NDIA SE Conference Progress Briefings• Comments on PPP related guidance
Schedule / Resources
• PPP Workshop Apr/May• Joint meetings with SED Committees Jun/Aug
• Developmental Test and Evaluation• Systems of Systems
Issues / Concerns:
• Industry and government engagement
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
52TOC
Summary of SSE Committee2014 Plans
Topic Activity
SSE
PPP Workshop 2014: Follow-on to 2012 Workshop Focus on Taxonomy and MetricsMay 20-22: MITRE facility in McLean, VA
Industry Inputs Comments on guideline documentsInputs into PPP implementation
SED
Systems of Systems 2014: PPP leverage points in the SoS Wave Model
Developmental Test and Evaluation
2014: Cyber testing guidelines connections to Program Protection Planning
Completed Current Proposed
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
53TOC
Systems Security Engineering
• Restart Former Systems Assurance Committee• New Systems Security Engineering Committee
• Kick-off June 18th
• Track at SE Symposium• Planning follow-on workshop in 2014 on Program Protection Plan
2013
NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013
54TOC
NDIA SE Conference2013
Issue Short Title NDIA SE Conference Paper
1 Taxonomy 16290 – Critical Program Information Test Vector (Geoff “Ninja” Donatelli, Raytheon)
2 Metrics 16185 – Software Assurance and NDAA 2013: Software Code Quality Checking (John Keane DoD VA IPO, Vik Chauhan Deloitte Consulting)
3 Contracts andAcquisition Strategy
16223 – System Security Engineering and Comprehensive Program Protection (Melinda Reed, OSASD SE)
4 Threat and Attack Vectors
16051 – Engineering Your Software for Attacks (Bob Martin, Mitre)16077 – Security Engineering in a Systems of Systems Environment (George Rebovich, Mitre)16001 – Strategic Cybersecurity Threat Analysis Framework: Know Your Enemy to Defeat Your Enemy (Michele Myauo, Microsoft)16111 – A Supply Chain Attack Framework to Support DoD Supply Chain Security Risk Management (Dr. John Miller, Mitre)
5 Education 16153 – A Practical Educational Approach to Program Protection Planning (Dr. Don Gelosh, Worcester Polytechnic Institute)