Upload
brandon-floyd
View
218
Download
1
Tags:
Embed Size (px)
Citation preview
1
CHALLENGES WITH QUANTIFYING THE QUALITATIVE
In collaboration with:
Elizabeth Whitaker, Erica Briscoe, Ethan Trewhitt, Georgia TechKevin Murphy, Frank Ritter, John Horgan, Penn State
Caroline Kennedy-Pipe, Univ. of Hull
Presented to:ONR Workshop on Human Interactions in Irregular Warfare as a Complex
System Atlanta, GA
13-14 April 2011
Presented by:Dr. Lora Weiss
Georgia Tech Research [email protected]
2
LINKING US-UK EXPERTISE(for understanding IED perpetration in Iraq)
SMEs
Doctrine
Literature
Scenarios
Knowledge Engineering
Influence ModelsSystem Dynamics
ModelsAgent-based Models
Models
Modeling
Evaluation
SMEs
Model Considerations
• Incomplete Data• Data Provenance• Data Uncertainty• Data Perishability
Objective• Provide a methodology to scientifically capture, evaluate, and predict large-scale behaviors of potential
IED developers before they have successfully deployed devices
• Elicit information from UK subject matter experts, who have had different experiences on their homeland
• Develop analytic tools to conduct quantitative and qualitative analysis of potential interdiction points
Individual
Individual
Individual
Individual
Individual
Individual
“Western”
“Non-Western”
Common Motivations ActivitiesCommon
Goals Results
Religion
Activities ResultsMoney
Power
Goal
Goal
Goal
4
Common End-state vs. Individual Motivations
Example Interview Results
• Management and planning within IED “teams” are different than in Western civilization• Participants are not necessarily focused on an end-state. Instead individual motivations
(that may differ) are manipulated toward the individual’s end goals.
5
Interview Results - 2
• Crucial differences:– IRA was aware they were being watched and operated in a manner to “fool” their
pursuers– Iraqi insurgency less of a “game-like” attitude and is more concentrated on purely
technical aspects
IRA
Counter-IED Monitoring
Iraqi Insurgency
Counter-IED Monitoring
Game-likeplanning
Little attempt toInfluence monitoring
Activities change by being monitored vs. concentrating on technical execution
6
Interview Results - 3
Normal Behavior
Current Behavior
Missing Normal
BehaviorUnusual
Anomalous Behavior
Comparison to spot differences
Record and Share
Stories
Useful information often lost because no explicit sharing of stories when units transfer
• This information is usually subtle and not directly transcribable, e.g., noticing what is not normal about an environment (social or physical)
• Military personnel notice things that are different and have a hard time putting their fingers on exactly what that is
7
Interview Results - 4
RecruitingPersonnel
Religious Motivation
Monetary Motivation
Power Motivation
Motivation varies among lower level participants
• For lower level participants (beneath management), motivation is most often monetary or peer involvement
• Experts are conflicted as to whether religion is actually a motivator or just used as ‘clean’ explanation
From Knowledge Engineering to Modeling
9
• Methodology for evaluating of complex systems over time • Represent causal relationships and feedback• Stocks and flows represent the movement of items, materials, people, or abstract
concepts • Easy experimentation with changes in structure, inputs, conditions
System Dynamics
11
Materials and Supplies
• Stock – Materials & Supplies - represents the inventory of generalized materials and supplies of insurgent groups in the area
• Input Flow - Gathering - represents actions that cause the accumulation of materials and supplies
• Output Flow - Consumption - represents the use of these materials and supplies in the construction of IEDs
12
IED Process
• Four stages of IED Process: Constructed, Inventory, Emplaced, Detonated
• Flows between stocks represent transitions from one stage to another
• The Disrupted IED stock and its related flows, Early, Middle and Late Disruption represent the destruction of IEDs by counter-IED efforts.
13
Personnel
• Represent the transition of a sympathizer into active participation within a terrorist group
• Radicalization represents transition of a person from within the general population to the Grey Population.
• A previously neutral person taking a position of sympathy for insurgent beliefs
• Deradicalization is the reverse of this, when a person loses sympathy for the insurgency
• As a person becomes an active participant in the IED process, this is represented as Recruitment
• Death and Disengagement indicate that an active insurgent has left the group in one way or another.
Two Radicalization Submodels
17
Based on method of Bartolomei, J., Casebeer, W., & Thomas, T. (2004)
Derived from SME input
Representing Culture Influences
• Complex socio-cultural computational models include both quantitative and qualitative data– Qualitative• Interviews with perpetrators• Opinions of SMEs• Broad social, psychological theories
– Quantitative• Demographics• Economic Factors• Surveys
18
Want to start understanding the interactions of all these influences What-If Analyses
Potential Events
Environmental Context
Scenarios
Models Evaluate
Impact of Change
What-if Analysis
Enable analysts to - Experiment with different sets of parameters, variables, and relationships- Explore results of
- Events within our control (military actions, policies, diplomatic decisions) - Events not within our control (weather, crop production, actions of others)
19
Modeling Approaches Across The Sciences
20
Figure from: G. Zacharias, J. MacMillan, and S. Van Hemel (eds), Behavioral Modeling and Simulation, National Research Council, 2008.
Qualitative Socio-Cultural Data
Psychological Theories
Cultural Descriptions
Political Attitudes,Influences
External Influences on Behaviors
Observed Behavior
PoliciesOpinions, Experiences of SMEs
Interviews with Individuals or Groups
Being ModeledSocial Theories
21
Quantitative Socio-Cultural Data
Survey Data
Census Data
Demographics
Economic FactorsEnvironmental Measurements
Psychometric Measurements
Polls
22
Representing Qualitative Data in a Computational Model
Representation ExamplesLandmark Values Left, Right, StraightLikert Values Strongly Disagree, Disagree, Neither Agree
nor Disagree, Agree, Strongly Agree“On a scale of 1 to 10 …”
Fuzzy Values Low, Medium, HighRelationships and Flows of Items, Attitudes, Information
Maturity Employment Stability
Decisions, Rules, CasesRelational Expressions, Equations
23
• Accepted measurement scales may not exist• Modelers may need to create fuzzy or landmark values for abstract concepts
What Kind of Data Does Your Model Need?
• What types of socio-cultural data does your model need?• What would you do if you never got it?• What are realistic substitutes and
workarounds?
24
Realistic Substitutes and Workarounds
Indirect ways to get at the information, perhaps with a
little more uncertainty
Other types of data that might be available to
substitute
Creation of data through laboratory experiments, and
synthetic data created by software generators
Perhaps a SME’s opinion included in the model would provide useful information if the data remains unavailable
Can we estimate the validity of this kind of surrogate data for use in a particular model?
25
Dealing with Uncertainty in Socio-Cultural Data
• Where does uncertainty occur?– Uncertainty in descriptions of attitudes, cultures, behaviors– Uncertainty in measurements (physical measurements or
survey instruments)– Uncertainty in descriptions of historical situations– Uncertainty inherent in human behavior
• Variations in human choices given the same culture and situation
• What approaches exist for dealing with uncertainty?– Probabilistic approaches, random variables– Representations of likelihood (other than strict probability)– Techniques for combining certainty values
26
Data Provenance
• Provenance: Where did the data come from, who collected it, how was it collected, under what circumstances, and what was the context?– The model user should understand the provenance of data in order
to determine how appropriate it is for a particular use.– Data from an authoritative source is not automatically more useful
than data from an unreliable source.– Data known with a high degree of certainty may not be the data that
leads to recognition of unexpected behaviors.– Once a piece of information has been confirmed and ‘hits the news’,
it may no longer provide information that can be acted upon.– In contrast, rumors about conspiracies, although potentially false,
are sources of information that may allow intervention to prevent catastrophic events.
27
Data Perishability and Missing Data
• Perishability: How long will this data be valid?– The importance of knowing when to remove data from a
model and recognizing that behaviors change and adapt
• Model representations in this space need to be those that can be made robust against missing features.
28
Summary
Develop separate federated
models
Different aspects of the domain
Different views (micro, meso,
macro)
Different time scales
Build iteratively
Allows insertion of new domain knowledge
Provides ability to change as your
knowledge of the model changes
Allows for correction and
results in better models
Build to allow for easy
extension
New domain knowledge
Changes in the system being
modeled
Changes in the intended use of
the model
Make limitations
explicit
Models are built with simplifying
assumptions
Based on the view or interpretation
of a modeler
Based on data or knowledge with
some level of uncertainty
Adopt Best Practices for Integrating Qualitative Data into Quantitative Models