P1516.4: VV&A Overlay to the P1516.4: VV&A Overlay to the FEDEPFEDEP
20 September 200720 September 2007
Briefing for the VV&A SummitBriefing for the VV&A Summit
Simone YoungbloodSimone YoungbloodM&S CO VV&A Proponency Leader
Current VV&A Standards InitiativesCurrent VV&A Standards Initiatives
• VV&A FEDEP Overlay
• VV&A Templates
BackgroundBackground
• VV&A Product Development Group (PDG) initiated in Spring 2004• VV&A Overlay Drafting Group had first meeting in Winter 2004• Draft overlay standard development progress briefed in Spring 2004,
Fall 2004, Spring 2005, Summer 2005 and Fall 2005• Draft overlay standard released to the VV&A PDG for comment on 31
March - 15 May 2006• Drafting Group met to develop comment resolution recommendations
on 5-9 June 2006• VV&A PDG met to approve comment resolutions on 28 June 2006• Revised draft standard released to the VV&A PDG for comment on
4 August - 4 September 2006• Drafting Group met to develop comment resolution recommendations
on 5-9 September 2006• VV&A PDG meets to decide recommended comment resolutions on
14 September 2006
ScopeScope
• Identifies and describes the recommended verification, validation and accreditation (VV&A) processes that one should perform to assure the acceptability and utility of federations for particular purposes.
• Identifies and describes the information feeding and resulting from those processes as well as the relationships between the High Level Architecture (HLA) Federation Development and Execution Process (FEDEP) and the VV&A processes and their respective information products .
• Defines those terms uniquely needed to characterize the FEDEP VV&A overlay (e.g., federation conceptual model, interoperability, representational requirements).
• Takes special care to use and build upon existing standards, standard terms and their definitions whenever possible.
• Focuses upon the VV&A processes that apply to federations.
• Does not address the VV&A processes associated with individual simulations (federates) but does consider using the information produced by those processes.
• Does not describe individual techniques that one might employ to execute the VV&A processes for federations.
Draft VV&A Overlay Standard OutlineDraft VV&A Overlay Standard Outline
• Introduction• Scope• Purpose• References• Definitions, Abbreviations &
Acronyms– Definitions– Abbreviations & Acronyms
• Roles & Responsibilities– Introduction to VV&A Roles &
Responsibilities– Role Mapping to Overlay
Activities
• Assumptions & Tailoring– Assumptions– Tailoring
• Overlay: Top Level View• Overlay: Detailed View
– Phase 1 - Verify Federation Objectives
– Phase 2 - V&V Federation Conceptual Model
– Phase 3 - Verify Federation Design
– Phase 4 - Verify Federation Development Products
– Phase 5 - Validate & Accept Federation
– Phase 6 - V&V Federation Outputs
– Phase 7 - Consolidate Federation VV&A Products
• Conclusion
Some Terms Important to the VV&A OverlaySome Terms Important to the VV&A Overlay
The drafting committee encountered several terms while drafting the VV&A Overlay to the FEDEP that were relevant to federation VV&A:
The drafting committee took special care to use the definitions of terms in the FEDEP and the well respected definitions from other sources whenever possible. They only judiciously crafted new definitions when no existing definition captured the meaning of the terms as they were used in the overlay (marked by an *).
• interoperability
• parameter
• referent
• representational requirements
• results sampling strategy *
• risk
• scenario
• substantive interoperability
• uncertainty *
• traceability
• validation
• verification
• acceptability criteria *
• acceptance *
• accreditation
• credibility *
• error characteristics *
• federate
• federation
• federation conceptual model
• federation objectives
• federation object model
• federation requirements *
• fidelity *
Overview of the Draft Overlay ActivitiesOverview of the Draft Overlay Activities
Phase 1 - Verify Federation Objectives Activity 1.1 - Support Identifying User/Sponsor
Needs
Activity 1.2 - Plan Accreditation Activities
Activity 1.3 - Support Developing Federation Objectives
Activity 1.4 - Contribute to Verifying Federation Objectives
Activity 1.5 - Assemble Federation Referent
Activity 1.6 - Define Federation Acceptability Criteria
Activity 1.7 - Plan V&V Activities
Phase 2 - V&V Conceptual Model Activity 2.1 - Support Developing Federation
Scenarios
Activity 2.2 - Contribute to Verifying Federation Scenarios
Activity 2.3 - Support Developing Federation Conceptual Model
Activity 2.4 - Contribute to Verifying Federation Conceptual Model
Activity 2.5 - Validate Federation Conceptual Model
Activity 2.6 - Support Developing Federation Requirements
Activity 2.7 - Contribute to Verifying Federation Requirements
Phase 3 - Verify Federation Design Activity 3.1 - Support Selecting Federates
Activity 3.2 - Support Preparing Federation Design
Activity 3.3 - Contribute to Verifying Federation Design
Activity 3.4 - Support Preparing Federation Development and Execution Plan
Phase 4 - Verify Federation Development Products Activity 4.1 - Support Developing FOM Activity 4.2 - Contribute to Verifying FOM Activity 4.3 - Support Establishing Federation
Agreements Activity 4.4 - Contribute to Verifying Federation
Agreements Activity 4.5 - Support Implementing Federate
Designs Activity 4.6 - Support Implementing Federation
Infrastructure Activity 4.7 - Contribute to Verifying Federation
Infrastructure Activity 4.8 - V&V Federation Data Sets
Phase 5 - Validate & Accept Federation Activity 5.1 - Support Planning Federation
Execution Activity 5.2 - Support Integrating Federation Activity 5.3 - Contribute to Verifying Integrated
Federation Activity 5.4 - Support Testing Federation Activity 5.5 - Validate Integrated Federation
Results Activity 5.6 - Perform Acceptance Assessment
Phase 6 - V&V Federation Outputs Activity 6.1 - Support Executing Federation Activity 6.2 - Contribute to Verifying Raw
Execution & Derived Output Activity 6.3 - Validate Federation Outputs
Phase 7 - Consolidate Federation VV&A Products Activity 7.1 - Support Analyzing Data Activity 7.2 - Prepare Federation VV&A Products
for Reuse
Top Level View of the VV&A Overlay to the Top Level View of the VV&A Overlay to the FEDEPFEDEP
FEDEP
VV&A Overlay to the FEDEP
FEDEP INFORMATION & PRODUCTS
Plan,Integrate
& TestFederation
Plan,Integrate
& TestFederation
Step 5
Validate &Accept
Federation
Validate &Accept
Federation
Phase 5
• Validate the integrated federation results against the Federation Acceptability Criteria and against the Federation Referent.
• Formulate federation acceptance/ accreditation recommen-dations.
ExecuteFederation& PrepareOutputs
ExecuteFederation& PrepareOutputs
Step 6
V&VFederation
Output
V&VFederation
Output
Phase 6
• Verify the federation outputs internally.
• Validate federation output against the Federation Acceptability Criteria and against the Federation Referent.
DefineFederationObjectives
DefineFederationObjectives
Verify Federation Objectives
Verify Federation Objectives
Phase 1
• Verify the Federation Objectives with the User /Sponsor.
• Define the Federation Acceptability Criteria.
• Assemble the Federation Referent.
• Plan the VV&A activities.
Step 1
PerformConceptual
Analysis
PerformConceptual
Analysis
V&VFederationConceptual
Model
V&VFederationConceptual
Model
Phase 2
• Verify the Federation Conceptual Model internally.
• Validate the Federation Conceptual Model against the Federation Referent and against the Federation Acceptability Criteria.
Step 2
DesignFederation
DesignFederation
VerifyFederation
Design
VerifyFederation
Design
Phase 3
• Verify the Federation Design internally.
• Verify the Federation Design against the Federation Conceptual Model.
Step 3
DevelopFederation
DevelopFederation
VerifyFederation
DevelopmentProducts
VerifyFederation
DevelopmentProducts
Phase 4
• Verify the federation development products internally.
• Verify the federation development products against the Federation Conceptual Model.
Step 4
AnalyzeData &
EvaluateResults
AnalyzeData &
EvaluateResults
ConsolidateFederation
VV&AProducts
ConsolidateFederation
VV&AProducts
Phase 7
• Collect all VV&A products and lessons learned.
• Assemble them into a VV&A package to support federation reuse.
Step 7
Detailed View of the VV&A Overlay to the Detailed View of the VV&A Overlay to the FEDEPFEDEP
Federation VV&A
products
Define Federation Objectives
Perform Conceptual Analysis
DevelopFederation
Plan, Integrate and Test Federation
Execute Federation and Prepare Outputs
Analyze Data and Evaluate Results
DesignFederation
verified Federation Objectives
validated Conceptual Model
verified Federation
design
verified Federation development
products
accepted Federation
validated Federation output
Verify Federation Objectives
1.0 V&V Federation Conceptual
Model2.0 Verify
Federation Design
3.0
Validate & Accept
Federation
5.0 V&V Federation
Output
6.0 Consolidate Federation
VV&A Products
7.0
Verify Federation
Development Products
4.0
federation output V&V
results
federation development verification results
federation validation and acceptance results
federation conceptual model V&V results
Federation Acceptability Criteria, Federation Referent, VV&A Plans
federation design verification results
Tasks Associated with Each VV&A Overlay Tasks Associated with Each VV&A Overlay PhasePhase
Verify Federation
Design
Verify Federation
Design
3.0
Support Selecting Federates
Support Preparing Federation
Design
Contribute to Verifying
Federation Design
Support Preparing Federation
Development & Execution
Plan
Verify Federation Objectives
Verify Federation Objectives
1.0
Support Identifying
User/Sponsor Needs
Plan Accreditation
Activities
Contribute to Verifying
Federation Objectives
Support Developing Federation Objectives
Assemble Federation Referent
Define Federation
Acceptability Criteria
Plan V&V Activities
V&V Federation Conceptual
Model
V&V Federation Conceptual
Model
2.0
Support Developing Federation Scenarios
Contribute to Verifying
Federation Scenarios
Contribute to Verifying
Federation Requirements
Support Developing Federation
Requirements
Support Developing Federation Conceptual
Model
Contribute to Verifying
Federation Conceptual
Model
Validate Federation Conceptual
Model
Validate & Accept
Federation
Validate & Accept
Federation
5.0
Support Planning
Federation Execution
Support Integrating Federation
Contribute to Verifying Integrated Federation
Support Testing
Federation
Perform Acceptance Assessment
Validate Integrated Federation
Results
V&V Federation
Output
V&V Federation
Output
6.0
Support Executing Federation
Contribute to Verifying Raw
Execution and Derived
Output
Validate Federation
Outputs
Consolidate Federation
VV&A Products
Consolidate Federation
VV&A Products
7.0
Support Analyzing
Data
Prepare Federation
VV&A Products for
Reuse
V&V Federation Data Sets
Verify Federation
Development Products
Verify Federation
Development Products
4.0
Support Developing
FOM
Contribute to Verifying
FOM
Support Establishing Federation
Agreements
Contribute to Verifying
Federation Agreements
Contribute to Verifying
Federation Infrastructure
Support Implementing
Federate Designs
Support Implementing
Federation Infrastructure
Phase 1 - Verify Federation ObjectivesPhase 1 - Verify Federation Objectives
Develop ObjectivesIdentify User/Sponsor
Needsverified Federation Objectives
Federation Acceptability
Criteria
Federation Referent
Federation V&V Plan
Federation Accreditation Plan
User/Sponsor Needs
SupportIdentifying User/Sponsor
Needs1.1
Plan Accreditation
Activities
1.2
Support Developing Federation
Objectives1.3
Contribute to Verifying Federation
Objectives1.4
Assemble Federation Referent
1.5
Define Federation
Acceptability Criteria
1.6
Plan V&V Activities
1.7
Phase 2 - V&V Federation Conceptual ModelPhase 2 - V&V Federation Conceptual Model
Develop Federation Requirements
verified Federation Scenarios
verified Federation Objectives
verified Federation
Requirements
validated Federation Conceptual Model
• Federation Referent• Federation Acceptability Criteria
Support Developing Federation Scenarios
2.1
Contribute to Verifying Federation
Scenarios2.2
Support Developing Federation
Conceptual Model
2.3
Contribute to Verifying Federation Conceptual Model
2.4
Validate Federation Conceptual
Model2.5
Contribute to Verifying Federation
Requirements
2.7
Support Developing Federation
Requirements
2.6
Develop Scenario
Develop Federation Conceptual Model
Phase 3 - Verify Federation DesignPhase 3 - Verify Federation Design
Prepare Plan (FDEP)
Select FederatesPrepare Federation
Design
Federation Acceptability Criteria
validated Federation
Conceptual Model
updated FDEP
• Federation Accreditation Plan• Federation V&V Plan
verified Federation design
Support Selecting Federates
3.1
Support Preparing Federation Design
3.2
Contribute to Verifying Federation
Design3.3
Support Preparing Federation
Development & Execution Plans
3.4
Phase 4 - Verify Federation Development Phase 4 - Verify Federation Development ProductsProducts
Support Establishing Federation
Agreements
Support Developing FOM
Support Implementing
Federate Designs
Develop FOMEstablish Federation
AgreementsImplement Federate
Designs
Data Sets
Support Implementing Federation
Infrastructure
Implement Federation Infrastructure
• verified federation design products• validated Federation Conceptual Model
existing federate data
sets
verified scenarios
verified FOM
verified Federation
Agreementsvalidated federates
verified federation
infrastructure
supporting
databases
validated data sets
• Federation Referent• Federation
Acceptability Criteria
4.1 4.3 4.5 4.6
Contribute to Verifying Federation
Agreements4.4
Contribute to Verifying Federation
Infrastructure4.7
Verify and Validate Data Sets
4.8
Contribute to Verifying FOM
4.2
Phase 5 - Validate and Accept FederationPhase 5 - Validate and Accept Federation
Plan Execution
Federation Referent
Federation Acceptability Criteria
V&V activity results
verified integrated federation
acceptance recommendation
validated Federation
Federation validation
results
Integrate Federation
Test Federation
Support Planning Federation Execution
5.1
Support Integrating Federation
5.2
Contribute to Verifying Integrated
Federation
5.3
Support Testing Federation
5.4
Validate Integrated Federation
Results5.5
Perform Acceptance Assessment
5.6
Phase 6 - V&V Federation OutputPhase 6 - V&V Federation Output
Execute FederationPrepare Federation
Outputs
• Federation Referent• Federation Acceptability
Criteria
federation output
validation results
federation outputs
Validate Federation
Outputs
6.3
Support Executing Federation
6.1
6.2
Contribute to Verifying Raw Execution and Derived Output
Phase 7 - Consolidate Federation VV&A Phase 7 - Consolidate Federation VV&A ProductsProducts
Analyze DataEvaluate and
Feedback Results
federation outputs
V&V activity products
analyzed data
reusable VV&A products
Support Analyzing Data
7.1
Prepare Federation
VV&A Products for Reuse
7.2
P1516.4 P1516.4 Recommended Practice for VV&A of a Federation, an Overlay to the HLA Recommended Practice for VV&A of a Federation, an Overlay to the HLA
Federation Development and Execution ProcessFederation Development and Execution Process
Initial BallotInitial Ballot
Ballot Open Date: 04/17/2007 Ballot Close Date: 05/17/2007
RESPONSE RATEThis ballot has met the 75% returned ballot requirement55 eligible people in this ballot group.44 affirmative votes3 negative votes with comments0 negative votes without comments5 abstention votes52 votes received = 95% returned, 10% abstention
APPROVAL RATEThe 75% affirmation requirement is being met44 affirmative votes3 negative votes with comments47 votes = 94% affirmative
P1516.4 P1516.4 Recommended Practice for VV&A of a Federation, an Overlay to Recommended Practice for VV&A of a Federation, an Overlay to
the HLA Federation Development and Execution Processthe HLA Federation Development and Execution Process
Recirculation BallotRecirculation Ballot
Ballot Open Date: 06/26/2007 Ballot Close Date: 07/06/2007
RESPONSE RATEThis ballot has met the 75% returned ballot requirement55 eligible people in this ballot group.45 affirmative votes2 negative votes with comments0 negative votes without comments5 abstention votes52 votes received = 95% returned, 10% abstention
APPROVAL RATEThe 75% affirmation requirement is being met.45 affirmative votes2 negative votes with comments47 votes = 96% affirmative
Summary of AccomplishmentsSummary of Accomplishments
• Developed a succinct and consistent model of the federation VV&A process that guided the the overlay’s formulation
• Crafted definitions for several important terms (e.g., federation requirements, credibility) where no relevant definitions existed in other sources
• Integrated the results from the preliminary NATO VV&A Overlay with the initial draft of the SISO VV&A Overlay to the FEDEP
• Clearly distinguished three important classes of tasks that involve the VV&A Team:
– Support - FEDEP tasks in which the VV&A Team should participate
– Contribute - verification tasks that the VV&A Team should monitor and supplement where necessary
– VV&A unique - tasks for which the VV&A Team are completely responsible
• Clarified the VV&A responsibilities for Steps 6 & 7 of the FEDEP, the life after federation acceptance
• Reached agreement between international representatives (Canada, France, Germany, Sweden, United Kingdom & USA) on the structure and details of the VV&A overlay to the FEDEP
• Ballotted P1516.4 through the IEEE process
Mission StatementMission Statement
Enable expanded M&S reuse by building the foundation for consistent V&V information to support accreditation decisions.
Outlines of the Four Core VV&A DocumentsOutlines of the Four Core VV&A Documents
Accreditation Plan V&V Plan V&V Report Accreditation Report
Executive Summary Executive Summary Executive Summary Executive Summary
1 Problem Statement 1 Problem Statement 1 Problem Statement 1 Problem Statement
2 M&S Requirements and Acceptability Criteria
2 M&S Requirements and Acceptability Criteria
2 M&S Requirements and Acceptability Criteria
2 M&S Requirements and Acceptability Criteria
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
4 Accreditation Methodology 4 V&V Methodology 4 V&V Task Analysis 4 Accreditation Assessment
5 Accreditation Issues 5 V&V Issues 5 V&V Recommendations 5 Accreditation Recommendations
6 Key Participants 6 Key Participants 6 Key Participants 6 Key Participants
7 Planned Accreditation Resources
7 Planned V&V Resources 7 Actual V&V Resources 7 Actual Accreditation Resources
8 V&V Lessons Learned 8 Accreditation Lessons Learned
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of ComparisonD ReferencesE Acronyms F GlossaryG Accreditation ProgrammaticsH Distribution List
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of Comparison D ReferencesE AcronymsF GlossaryG V&V ProgrammaticsH Distribution ListI Accreditation Plan
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of ComparisonD ReferencesE AcronymsF GlossaryG V&V ProgrammaticsH Distribution ListI V&V PlanJ Test Information
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of Comparison D ReferencesE AcronymsF GlossaryG Accreditation ProgrammaticsH Distribution ListI Accreditation PlanJ V&V Report