P1516.4: VV&A Overlay to the FEDEP 20 September 2007 Briefing for the VV&A Summit

P1516.4: VV&A Overlay to the P1516.4: VV&A Overlay to the FEDEPFEDEP

20 September 200720 September 2007

Briefing for the VV&A SummitBriefing for the VV&A Summit

Simone YoungbloodSimone YoungbloodM&S CO VV&A Proponency Leader

[email protected]

Current VV&A Standards InitiativesCurrent VV&A Standards Initiatives

• VV&A FEDEP Overlay

• VV&A Templates

VV&A FEDEP OverlayVV&A FEDEP Overlay

BackgroundBackground

• VV&A Product Development Group (PDG) initiated in Spring 2004• VV&A Overlay Drafting Group had first meeting in Winter 2004• Draft overlay standard development progress briefed in Spring 2004,

Fall 2004, Spring 2005, Summer 2005 and Fall 2005• Draft overlay standard released to the VV&A PDG for comment on 31

March - 15 May 2006• Drafting Group met to develop comment resolution recommendations

on 5-9 June 2006• VV&A PDG met to approve comment resolutions on 28 June 2006• Revised draft standard released to the VV&A PDG for comment on

4 August - 4 September 2006• Drafting Group met to develop comment resolution recommendations

on 5-9 September 2006• VV&A PDG meets to decide recommended comment resolutions on

14 September 2006

ScopeScope

• Identifies and describes the recommended verification, validation and accreditation (VV&A) processes that one should perform to assure the acceptability and utility of federations for particular purposes.

• Identifies and describes the information feeding and resulting from those processes as well as the relationships between the High Level Architecture (HLA) Federation Development and Execution Process (FEDEP) and the VV&A processes and their respective information products .

• Defines those terms uniquely needed to characterize the FEDEP VV&A overlay (e.g., federation conceptual model, interoperability, representational requirements).

• Takes special care to use and build upon existing standards, standard terms and their definitions whenever possible.

• Focuses upon the VV&A processes that apply to federations.

• Does not address the VV&A processes associated with individual simulations (federates) but does consider using the information produced by those processes.

• Does not describe individual techniques that one might employ to execute the VV&A processes for federations.

Draft VV&A Overlay Standard OutlineDraft VV&A Overlay Standard Outline

• Introduction• Scope• Purpose• References• Definitions, Abbreviations &

Acronyms– Definitions– Abbreviations & Acronyms

• Roles & Responsibilities– Introduction to VV&A Roles &

Responsibilities– Role Mapping to Overlay

Activities

• Assumptions & Tailoring– Assumptions– Tailoring

• Overlay: Top Level View• Overlay: Detailed View

– Phase 1 - Verify Federation Objectives

– Phase 2 - V&V Federation Conceptual Model

– Phase 3 - Verify Federation Design

– Phase 4 - Verify Federation Development Products

– Phase 5 - Validate & Accept Federation

– Phase 6 - V&V Federation Outputs

– Phase 7 - Consolidate Federation VV&A Products

• Conclusion

Some Terms Important to the VV&A OverlaySome Terms Important to the VV&A Overlay

The drafting committee encountered several terms while drafting the VV&A Overlay to the FEDEP that were relevant to federation VV&A:

The drafting committee took special care to use the definitions of terms in the FEDEP and the well respected definitions from other sources whenever possible. They only judiciously crafted new definitions when no existing definition captured the meaning of the terms as they were used in the overlay (marked by an *).

• interoperability

• parameter

• referent

• representational requirements

• results sampling strategy *

• risk

• scenario

• substantive interoperability

• uncertainty *

• traceability

• validation

• verification

• acceptability criteria *

• acceptance *

• accreditation

• credibility *

• error characteristics *

• federate

• federation

• federation conceptual model

• federation objectives

• federation object model

• federation requirements *

• fidelity *

Overview of the Draft Overlay ActivitiesOverview of the Draft Overlay Activities

Phase 1 - Verify Federation Objectives Activity 1.1 - Support Identifying User/Sponsor

Needs

Activity 1.2 - Plan Accreditation Activities

Activity 1.3 - Support Developing Federation Objectives

Activity 1.4 - Contribute to Verifying Federation Objectives

Activity 1.5 - Assemble Federation Referent

Activity 1.6 - Define Federation Acceptability Criteria

Activity 1.7 - Plan V&V Activities

Phase 2 - V&V Conceptual Model Activity 2.1 - Support Developing Federation

Scenarios

Activity 2.2 - Contribute to Verifying Federation Scenarios

Activity 2.3 - Support Developing Federation Conceptual Model

Activity 2.4 - Contribute to Verifying Federation Conceptual Model

Activity 2.5 - Validate Federation Conceptual Model

Activity 2.6 - Support Developing Federation Requirements

Activity 2.7 - Contribute to Verifying Federation Requirements

Phase 3 - Verify Federation Design Activity 3.1 - Support Selecting Federates

Activity 3.2 - Support Preparing Federation Design

Activity 3.3 - Contribute to Verifying Federation Design

Activity 3.4 - Support Preparing Federation Development and Execution Plan

Phase 4 - Verify Federation Development Products Activity 4.1 - Support Developing FOM Activity 4.2 - Contribute to Verifying FOM Activity 4.3 - Support Establishing Federation

Agreements Activity 4.4 - Contribute to Verifying Federation

Agreements Activity 4.5 - Support Implementing Federate

Designs Activity 4.6 - Support Implementing Federation

Infrastructure Activity 4.7 - Contribute to Verifying Federation

Infrastructure Activity 4.8 - V&V Federation Data Sets

Phase 5 - Validate & Accept Federation Activity 5.1 - Support Planning Federation

Execution Activity 5.2 - Support Integrating Federation Activity 5.3 - Contribute to Verifying Integrated

Federation Activity 5.4 - Support Testing Federation Activity 5.5 - Validate Integrated Federation

Results Activity 5.6 - Perform Acceptance Assessment

Phase 6 - V&V Federation Outputs Activity 6.1 - Support Executing Federation Activity 6.2 - Contribute to Verifying Raw

Execution & Derived Output Activity 6.3 - Validate Federation Outputs

Phase 7 - Consolidate Federation VV&A Products Activity 7.1 - Support Analyzing Data Activity 7.2 - Prepare Federation VV&A Products

for Reuse

Top Level View of the VV&A Overlay to the Top Level View of the VV&A Overlay to the FEDEPFEDEP

FEDEP

VV&A Overlay to the FEDEP

FEDEP INFORMATION & PRODUCTS

Plan,Integrate

& TestFederation

Plan,Integrate

& TestFederation

Step 5

Validate &Accept

Federation

Validate &Accept

Federation

Phase 5

• Validate the integrated federation results against the Federation Acceptability Criteria and against the Federation Referent.

• Formulate federation acceptance/ accreditation recommen-dations.

ExecuteFederation& PrepareOutputs

ExecuteFederation& PrepareOutputs

Step 6

V&VFederation

Output

V&VFederation

Output

Phase 6

• Verify the federation outputs internally.

• Validate federation output against the Federation Acceptability Criteria and against the Federation Referent.

DefineFederationObjectives

DefineFederationObjectives

Verify Federation Objectives


Phase 1

• Verify the Federation Objectives with the User /Sponsor.

• Define the Federation Acceptability Criteria.

• Assemble the Federation Referent.

• Plan the VV&A activities.

Step 1

PerformConceptual

Analysis

PerformConceptual

Analysis

V&VFederationConceptual

Model

V&VFederationConceptual

Model

Phase 2

• Verify the Federation Conceptual Model internally.

• Validate the Federation Conceptual Model against the Federation Referent and against the Federation Acceptability Criteria.

Step 2

DesignFederation

DesignFederation

VerifyFederation

Design

VerifyFederation

Design

Phase 3

• Verify the Federation Design internally.

• Verify the Federation Design against the Federation Conceptual Model.

Step 3

DevelopFederation

DevelopFederation

VerifyFederation

DevelopmentProducts

VerifyFederation

DevelopmentProducts

Phase 4

• Verify the federation development products internally.

• Verify the federation development products against the Federation Conceptual Model.

Step 4

AnalyzeData &

EvaluateResults

AnalyzeData &

EvaluateResults

ConsolidateFederation

VV&AProducts

ConsolidateFederation

VV&AProducts

Phase 7

• Collect all VV&A products and lessons learned.

• Assemble them into a VV&A package to support federation reuse.

Step 7

Detailed View of the VV&A Overlay to the Detailed View of the VV&A Overlay to the FEDEPFEDEP

Federation VV&A

products

Define Federation Objectives

Perform Conceptual Analysis

DevelopFederation

Plan, Integrate and Test Federation

Execute Federation and Prepare Outputs

Analyze Data and Evaluate Results

DesignFederation

verified Federation Objectives

validated Conceptual Model

verified Federation

design

verified Federation development

products

accepted Federation

validated Federation output


1.0 V&V Federation Conceptual

Model2.0 Verify

Federation Design

3.0

Validate & Accept

Federation

5.0 V&V Federation

Output

6.0 Consolidate Federation

VV&A Products

7.0

Verify Federation

Development Products

4.0

federation output V&V

results

federation development verification results

federation validation and acceptance results

federation conceptual model V&V results

Federation Acceptability Criteria, Federation Referent, VV&A Plans

federation design verification results

Tasks Associated with Each VV&A Overlay Tasks Associated with Each VV&A Overlay PhasePhase

Verify Federation

Design

Verify Federation

Design

3.0

Support Selecting Federates

Support Preparing Federation

Design

Contribute to Verifying

Federation Design


Development & Execution

Plan



1.0

Support Identifying

User/Sponsor Needs

Plan Accreditation

Activities


Federation Objectives

Support Developing Federation Objectives

Assemble Federation Referent

Define Federation

Acceptability Criteria

Plan V&V Activities

V&V Federation Conceptual

Model

V&V Federation Conceptual

Model

2.0

Support Developing Federation Scenarios


Federation Scenarios


Federation Requirements

Support Developing Federation

Requirements

Support Developing Federation Conceptual

Model


Federation Conceptual

Model

Validate Federation Conceptual

Model

Validate & Accept

Federation

Validate & Accept

Federation

5.0

Support Planning

Federation Execution

Support Integrating Federation

Contribute to Verifying Integrated Federation

Support Testing

Federation

Perform Acceptance Assessment

Validate Integrated Federation

Results

V&V Federation

Output

V&V Federation

Output

6.0

Support Executing Federation

Contribute to Verifying Raw

Execution and Derived

Output

Validate Federation

Outputs

Consolidate Federation

VV&A Products

Consolidate Federation

VV&A Products

7.0

Support Analyzing

Data

Prepare Federation

VV&A Products for

Reuse

V&V Federation Data Sets

Verify Federation


Verify Federation


4.0

Support Developing

FOM


FOM

Support Establishing Federation

Agreements


Federation Agreements


Federation Infrastructure

Support Implementing

Federate Designs


Federation Infrastructure

Phase 1 - Verify Federation ObjectivesPhase 1 - Verify Federation Objectives

Develop ObjectivesIdentify User/Sponsor

Needsverified Federation Objectives

Federation Acceptability

Criteria

Federation Referent

Federation V&V Plan

Federation Accreditation Plan

User/Sponsor Needs

SupportIdentifying User/Sponsor

Needs1.1

Plan Accreditation

Activities

1.2


Objectives1.3

Contribute to Verifying Federation

Objectives1.4

Assemble Federation Referent

1.5

Define Federation


1.6

Plan V&V Activities

1.7

Phase 2 - V&V Federation Conceptual ModelPhase 2 - V&V Federation Conceptual Model

Develop Federation Requirements

verified Federation Scenarios

verified Federation Objectives

verified Federation

Requirements

validated Federation Conceptual Model

• Federation Referent• Federation Acceptability Criteria

Support Developing Federation Scenarios

2.1


Scenarios2.2


Conceptual Model

2.3

Contribute to Verifying Federation Conceptual Model

2.4

Validate Federation Conceptual

Model2.5


Requirements

2.7


Requirements

2.6

Develop Scenario

Develop Federation Conceptual Model

Phase 3 - Verify Federation DesignPhase 3 - Verify Federation Design

Prepare Plan (FDEP)

Select FederatesPrepare Federation

Design

Federation Acceptability Criteria

validated Federation

Conceptual Model

updated FDEP

• Federation Accreditation Plan• Federation V&V Plan

verified Federation design

Support Selecting Federates

3.1

Support Preparing Federation Design

3.2


Design3.3


Development & Execution Plans

3.4

Phase 4 - Verify Federation Development Phase 4 - Verify Federation Development ProductsProducts

Support Establishing Federation

Agreements

Support Developing FOM


Federate Designs

Develop FOMEstablish Federation

AgreementsImplement Federate

Designs

Data Sets

Support Implementing Federation

Infrastructure

Implement Federation Infrastructure

• verified federation design products• validated Federation Conceptual Model

existing federate data

sets

verified scenarios

verified FOM

verified Federation

Agreementsvalidated federates

verified federation

infrastructure

supporting

databases

validated data sets

• Federation Referent• Federation


4.1 4.3 4.5 4.6


Agreements4.4


Infrastructure4.7

Verify and Validate Data Sets

4.8

Contribute to Verifying FOM

4.2

Phase 5 - Validate and Accept FederationPhase 5 - Validate and Accept Federation

Plan Execution

Federation Referent

Federation Acceptability Criteria

V&V activity results

verified integrated federation

acceptance recommendation

validated Federation

Federation validation

results

Integrate Federation

Test Federation

Support Planning Federation Execution

5.1

Support Integrating Federation

5.2

Contribute to Verifying Integrated

Federation

5.3

Support Testing Federation

5.4

Validate Integrated Federation

Results5.5

Perform Acceptance Assessment

5.6

Phase 6 - V&V Federation OutputPhase 6 - V&V Federation Output

Execute FederationPrepare Federation

Outputs

• Federation Referent• Federation Acceptability

Criteria

federation output

validation results

federation outputs

Validate Federation

Outputs

6.3

Support Executing Federation

6.1

6.2

Contribute to Verifying Raw Execution and Derived Output

Phase 7 - Consolidate Federation VV&A Phase 7 - Consolidate Federation VV&A ProductsProducts

Analyze DataEvaluate and

Feedback Results

federation outputs

V&V activity products

analyzed data

reusable VV&A products

Support Analyzing Data

7.1

Prepare Federation

VV&A Products for Reuse

7.2

P1516.4 P1516.4 Recommended Practice for VV&A of a Federation, an Overlay to the HLA Recommended Practice for VV&A of a Federation, an Overlay to the HLA

Federation Development and Execution ProcessFederation Development and Execution Process

Initial BallotInitial Ballot

Ballot Open Date: 04/17/2007 Ballot Close Date: 05/17/2007

RESPONSE RATEThis ballot has met the 75% returned ballot requirement55 eligible people in this ballot group.44 affirmative votes3 negative votes with comments0 negative votes without comments5 abstention votes52 votes received = 95% returned, 10% abstention

APPROVAL RATEThe 75% affirmation requirement is being met44 affirmative votes3 negative votes with comments47 votes = 94% affirmative

P1516.4 P1516.4 Recommended Practice for VV&A of a Federation, an Overlay to Recommended Practice for VV&A of a Federation, an Overlay to

the HLA Federation Development and Execution Processthe HLA Federation Development and Execution Process

Recirculation BallotRecirculation Ballot

Ballot Open Date: 06/26/2007 Ballot Close Date: 07/06/2007

RESPONSE RATEThis ballot has met the 75% returned ballot requirement55 eligible people in this ballot group.45 affirmative votes2 negative votes with comments0 negative votes without comments5 abstention votes52 votes received = 95% returned, 10% abstention

APPROVAL RATEThe 75% affirmation requirement is being met.45 affirmative votes2 negative votes with comments47 votes = 96% affirmative

Summary of AccomplishmentsSummary of Accomplishments

• Developed a succinct and consistent model of the federation VV&A process that guided the the overlay’s formulation

• Crafted definitions for several important terms (e.g., federation requirements, credibility) where no relevant definitions existed in other sources

• Integrated the results from the preliminary NATO VV&A Overlay with the initial draft of the SISO VV&A Overlay to the FEDEP

• Clearly distinguished three important classes of tasks that involve the VV&A Team:

– Support - FEDEP tasks in which the VV&A Team should participate

– Contribute - verification tasks that the VV&A Team should monitor and supplement where necessary

– VV&A unique - tasks for which the VV&A Team are completely responsible

• Clarified the VV&A responsibilities for Steps 6 & 7 of the FEDEP, the life after federation acceptance

• Reached agreement between international representatives (Canada, France, Germany, Sweden, United Kingdom & USA) on the structure and details of the VV&A overlay to the FEDEP

• Ballotted P1516.4 through the IEEE process

VV&A TemplatesVV&A Templates

Mission StatementMission Statement

Enable expanded M&S reuse by building the foundation for consistent V&V information to support accreditation decisions.

Outlines of the Four Core VV&A DocumentsOutlines of the Four Core VV&A Documents

Accreditation Plan V&V Plan V&V Report Accreditation Report

Executive Summary Executive Summary Executive Summary Executive Summary

1 Problem Statement 1 Problem Statement 1 Problem Statement 1 Problem Statement

2 M&S Requirements and Acceptability Criteria




3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts




4 Accreditation Methodology 4 V&V Methodology 4 V&V Task Analysis 4 Accreditation Assessment

5 Accreditation Issues 5 V&V Issues 5 V&V Recommendations 5 Accreditation Recommendations

6 Key Participants 6 Key Participants 6 Key Participants 6 Key Participants

7 Planned Accreditation Resources

7 Planned V&V Resources 7 Actual V&V Resources 7 Actual Accreditation Resources

8 V&V Lessons Learned 8 Accreditation Lessons Learned

Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of ComparisonD ReferencesE Acronyms F GlossaryG Accreditation ProgrammaticsH Distribution List

Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of Comparison D ReferencesE AcronymsF GlossaryG V&V ProgrammaticsH Distribution ListI Accreditation Plan

Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of ComparisonD ReferencesE AcronymsF GlossaryG V&V ProgrammaticsH Distribution ListI V&V PlanJ Test Information

Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of Comparison D ReferencesE AcronymsF GlossaryG Accreditation ProgrammaticsH Distribution ListI Accreditation PlanJ V&V Report

Templates StatusTemplates Status

• Draft MIL Standard entitled Department of Defense Standard Practice: Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A) Documentation Templates

• DSP review closed 17 July 07

• Comment adjudication completed 23 August 07