Irregular Warfare Modeling/Data Validation Best Practices

11th Annual MOVES Research Education Summit

12 July 2011

Purpose and Agenda

2

Purpose: To discuss the framework developed at TRAC-MTRY/NPS toward conducting validation of Methods, Models, and Tools (MMT)

that inform analysis of Irregular Warfare (IW) and to overview ongoing development of a similar data validation framework.

12 July 2011 11th MOVES Research and Education Summit

• Overview and Objectives.

• Validation Project Teams.

• Project Descriptions and Results.

−IW Model Validation Best Practices.

−IW Data Validation Best Practices.

• Summary and Way Ahead.

• Questions.

Overview and Objectives

12 July 2011 311th MOVES Research and Education Summit

• TRAC-MTRY and Naval Postgraduate School.

– IW Modeling Validation Best Practices Guide, FY09-FY10.

- Identify and develop a best practices for verifying and validating social science theory and modeling required for tactical level IW MMT.

– Validating Data that Supports IW Models, FY10-FY11.

- Assess, at a minimum, the validation of IW data, to include an examination of data requirements, data sources, and data availability as well as derivation of data.

Objective: To develop methodologies/best practices toward validating methods, models and tools (MMT) and associated data that support IW

analysis, using the CG model for implementation of these practices.

Validation Project Teams

12 July 2011 411th MOVES Research and Education Summit

• FY09/FY10 TRAC-MTRY/NPS.– CPT Richard F. Brown, Project Lead.

– Dr. Jeffrey Appleget, Professor, OR Department.

– Dr. Deborah Duong, Senior Analyst, Augustine Consulting, Inc.

– Mr. Curtis Blais, Professor, MOVES Institute.

– Mr. Timothy K. Perkins, ERDC.

– Ms. Meredith Thompson, SEED Center.

• FY10/FY11 TRAC-MTRY/NPS.– MAJ Joe Vargas, Project Lead.

– CPT Richard F. Brown, Project Support.

– Dr. Jeffrey Appleget, Professor, OR Department.

– Mr. Curtis Blais, Professor, MOVES Institute.

– Dr. Michael Jaye, Professor, Defense Analysis Department.

– Dr. Eric Weisel, CEO, WernerAnderson, Inc.

– COL Robert Burks, Senior Military Advisor, OR Department.

Each team represents efforts that focus on developing frameworks and methodologies toward validating models, validating data, and

making assessments of and building documentation toward validation of CG.

Best Practices GuideApproach and Requirements

• Requirement 1. Provide a report that identifies validation best practices for IW MMT tactical-level representation.

• Requirement 2. Apply the BPG developed in Requirement 1 to both relevant MMT in order to demonstrate usability of the BPG framework.

• Best Practice Guide Approach.

– Conducted an extensive literature review of prevalent validation literature, to include the DOD Recommended Best Practices, MIL-STD-3022, SISO General Methodology for VV&A, and others (Chapter 2).

– Synthesized literature and described the challenges of modeling the IW environment and validating the models that represent the IW environment (Chapter 3).

– Constructed a validation framework toward model validation. The framework includes detailed descriptions of the principal areas of the framework that must be identified to conduct proper validation (Chapter 4).

– Apply the BPG, as appropriate, to relevant MMT; in particular, the CG model.

12 July 2011 511th MOVES Research and Education Summit

Validation Framework Concept Map“The Golden Triangle”

This effort resulted in the construction of the following model validation and development framework. The components of the “Golden Triangle” make up the crux of

model validation and establish the critical basis of comparison to the real world.

Validation Framework Concept Map.

• User Needs– The developer needs to obtain

a succinct and clear statement of the problem the M&S is expected to address.

• Requirements– Develop specific functional or

quality statements that can be directly and explicitly assessed to determine requirement.

• Acceptability Criteria– Develop a requirements

traceability matrix relating each specified requirement with acceptability criteria applicable to the intended use.

• Results– The acceptability criteria

identify what the model needs to do to satisfy or meet the set of respective requirements pertinent to the intended use.

• Data– The greater the specificity in

the data requirements for a model, the greater the ability to collect the data needed to populate the model.

• Intended Use.– Obtain a clear, succinct

statement of intended use from the user representatives.

• Simuland.– The simuland is the real-world

system of interest, including the objects, processes, or phenomena to be simulated.

• Referent.– Identify the social science

theory (or theories, if multiple competing theories will be represented in the model for comparison) that explains that phenomena.

• Conceptual Model.– Develop the conceptual model

using tools and techniques that create machine-readable specifications of the data and logic of the model.

• Executable Model.– Design the model

implementation to be as transparent as possible to permit analysis of execution paths and computed outcomes

12 July 2011 611th MOVES Research and Education Summit

IW Data Validation Best Practices

12 July 2011 711th MOVES Research and Education Summit

WORKING OUTLINE

• Introduction.

• Literature Review.– Review validation standard and practices from DOD and related sources that add value to DoD

MMT.

• Data Supporting Irregular Warfare Modeling.– Describe IW-type data and highlight the challenges in validating this data.

• IW Data Sources.– Characterize sources for IW data.

• Best Practices for Validating IW Data.– Explain the IW data validation framework, to include a description of methods toward validating IW

data, what metadata must be included with data sets, and other significant insights.

• Summary and Recommendations.

• Appendix.– Develop IW data validation framework use-case using relevant MMT, in particular, the CG Model.

Requirement: Provide a report that assesses, at a minimum, the validation of IW data, to include an examination of data requirements,

data sources, and data availability as well as the derivation of data.

Summary and Way Ahead

12 July 2011 811th MOVES Research and Education Summit

• Summary.– Multi-disciplinary team developed a BPG that provides a useful framework

that a validation agent could use for validating a IW model, or that a developer could use to properly develop an IW model.

– Model validation best practices incorporate concepts that developers are aware of, yet many of the concepts must be enumerated explicitly to ensure validation is possible.

– “Validation” may not be a proper term with respect to data. A paradigm shift may be in order.

– Any type of validation, whether model or data, must be related to a specified and/or intended use.

• Way Ahead.– TRADOC Analysis Center (TRAC) Irregular Warfare Modeling and Simulation

Summit provided a considerable forum for discussion and actions pertaining to these issues. The validation team will work closely with the TRAC Complex Operations Data Development Activity (CODDA) to develop and vet the data validation best practices, and ensure that is codified sufficiently for use in the analytic community.

9

Comments / Discussion

12 July 2011 11th MOVES Research and Education Summit

CG Model Assessment Requirements

12 July 2011 1011th MOVES Research and Education Summit

• Assess the appropriateness of the CG model representation of the Conflict Ecosystem conceptual model, and whether the model referent provided appropriate underpinning to account for human socio-cultural behavior.

• Assess the appropriateness of the Social Network Representation within the CG Model and the whether this representation is generalizable to conflict ecosystems in any region.

• Assess the appropriateness of the representation of social science theory within the CG Model, specifically tracing the implementation of the Theory of Planned Behavior.

• Develop risk measures for using a model or simulation for a specified application.

• Develop criteria for assessing the risk of using a model or simulation based on consequence, error, and validation process.

• Characterize an approach toward validating the CG model based on the methodologies/insights/lessons learned from the tasks developed above.