Modeling Human Reasoning About Meta-Information

Presented By:Scott LangevinJingsong Wang

Introduction• Human decision-making in real-time, dynamic

environments is becoming more complex• Decision-makers must manage large amounts of

incoming information and integrate it with previous knowledge to develop a “situational awareness”

• Relies on domain-knowledge but also on the qualifiers (meta-information) describing the information

• Problem: To replicate human reasoning or behavior, need to model both information and meta-information

• Most approaches have focused on representing the information, but little discussion of the meta-information

What is Meta-Information?

• Definitions– Data is output from a system that may or may not be useful to

decision-making (radar reports storm is coming)– Information is recognized inputs that are useable to decision-

making (storm is coming that may affect UAVs)– Meta-data is qualifiers of data that may or may not be useful to

decision-making (radar can locate aircraft with error of +/-1.5m)– Meta-information is qualifiers of information that affect decision-

making, reasoning, or behavior• Information processing• Situation awareness • Decision-making

• Definitions serve to explicitly identify the critical role of meta-information in human decision-making

Human Behavioral Models• Attempt to replicate human cognitive processes• Attempt to model human behaviors must capture the

impact of meta-information• HBM have wide variety of applications

– Developing and testing theories of human cognition– Representing realistic human behavior in training– Expert and Decision Support Systems

• Modelers typically do not address meta-information because of challenges acquiring, aggregating and integrating

• Focus of this research is on modeling meta-information in Bayesian Belief Networks (BBNs)

Uncertainty and Human Decision-Making• Human decision-making under uncertainty deviates from logical

decision-making and largely based on experience-based heuristic methods

• Often the heuristics represent how experts reason about the meta-information

• Uncertainty of information is one type of meta-information• Different methods of classifying uncertainty:

– Executional uncertainty– Goal uncertainty– Environment uncertainty– Lack of information, etc

• While these classifications of uncertainty and an understanding of their impacts on decision-making have been useful, they may not generalize to other types of meta-information not based on uncertainty (recency, reliability, trust)

Computational Approaches to Uncertainty

• Probability Measures• Dempster–Shafer belief functions• Extensions to first-order logic (e.g., defeasible

reasoning, argumentation)• Ranking functions• ‘plausibility” measures• Fuzzy set theory• Causal network methods (e.g., Bayesian belief

networks, similarity networks, influence diagrams)

Types and Sources of Meta-Information

• Identified the main types of meta-information that impact the decision-making process

• Research from over 30 domain experts, and over 500h of interviews, observations and evaluations

• From this developed a list of sources and types of meta-information that was consistently encountered across application domains

• Believe this approach developed an understanding of expert reasoning and behavior sufficient to understand the impact of meta-information at a level that supports modeling

Types and Sources of Meta-Information

Modeling Human Reasoning and Behavior• Computational Representation of human reasoning and

behavior• Model based on recognition-primed decision-making

– Experts do not do significant amounts of reasoning and problem solving, but rather have been trained to recognize critical elements of a situation and act accordingly

– Domain independent, modeling situation awareness-centered decision-making in high-stress, time-critical environments

• SAMPLE is a general use HBM– Defined modules: Information Processing, Situation Assessment, Decision

Making– Inputs processed by information processing module– Processed data (detected events) passed to situation

assessment module– Assessed situation is passed to decision-making module

• Rules, or lookup table of actions after situational assessment performed

SAMPLE Model

Bayesian Modeling about and with Meta-Information

• Difficult aspect of modeling human cognition and behavioral processes is the need to reflect the known impacts of meta-information on those processes

• Identified five features of reasoning that need representation within human behavior models:– Should succeed or fail to recognize relevant meta-information based on

attentional and cognitive demands– Should support the representation of successful or unsuccessful human

strategies to process information according to meta-information– Should represent the aggregation of meta-information– Should capture how effectively meta-information is understood relative

to any prior understanding or knowledge– Should succeed and fail at incorporating meta-information-mediated

situation assessments into behavior or decisions

Methods for Representing Human Reasoning

• Bayesian belief networks• Fuzzy set theory• Rule-based production systems• Case-based reasoning

• BBNs address multiple types of modeling requirements

• Two types of meta-information reasoning– Deductive reasoning– Abductive reasoning

• BBNs support both types of reasoning

Two Types of Reasoning

Modeling the Recognition and Aggregation of Meta-Information

• In many cases, human decision-makers will have to compute meta-information from multiple factors

• Data and meta-data can map to meta-information in the following ways:– One-to-one mappings– Many-to-one mappings– One-to-many mappings– Many-to-many mappings

• Once meta-information is calculated, it can influence the information gathering, situation assessment, and decision-making process

Applying BBNs to Model Congnitive Computation of Meta Information

Sensor Type as Node in Network: Sensor Type 3

Sensor Type as Node in Network: Sensor Type 1

Aggregating Meta-Information to Compute Overall Confidence

Modeling the Recognition and Aggregation of Meta-Information

• Knowing the best means to aggregate meta-information is challenging– Observation and study of human decision-making amongst

subject matter experts may provide some justification, but will often unavoidably result in inclusion of biases

– Using engineering data about sources may not adequately represent how a human would reason about meta-information, resulting in less reflective human behavior models

Modeling the Impact of Meta-Information on Situation Assessment

• Three Approaches– Simply filter or prioritize information based on meta-

information– Include meta-information within BBN models of

information gathering, situation assessment, and decision-making processes

– Use the meta-information in a specific parameter

Incorporating Meta-Information Explicitly into a BBN: No Confidence

Incorporating Meta-Information Explicitly into a BBN: Low Confidence

Examples of Computing the Probability of a Discrete Value for a BBN Node

Conclusion

• We described the application of meta-information and BBNs in modeling each of the following types of cognitive tasks:– Recognition of relevant meta-information based on aggregation of

available data, meta-data, information, and meta-information into types of meta-information.

– Filtering and prioritization of information based on meta-information.

– Aggregation of different types of meta-information to acquire their combined impact.

– Understanding of the impact of meta-information on existing knowledge

– Incorporation of meta-information into mediation of situation assessment and decision-making.

Questions?