Bisantz and Michael Jenkins, University at Buffalonagi/MURI/MURI/Year_2_files... · 2011. 9. 26. · Human Source Characterization & Human in the Fusion Loop Ann Bisantz and Michael

Human Source Characterization & Human in the Fusion Loop

Ann Bisantz and Michael Jenkins, University at BuffaloTel. (716) 645‐4714, E‐Mail: [email protected]

Objectives• Expand on source characterization efforts to include additional categories

• Gain an understanding of the cognitive requirements, complexities and constraints of the intelligence analysis (IA) domain

• Identify key human‐fusion system interaction touch points for the MURI system

Scientific/Technical Approach• Literature based review of existing cognitive research in IA

• Adaptation of proven HF techniques to develop method for touch point identification

• Application of developed method to MURI fusion system

Accomplishments• Characterized 8 additional soft observation categories

• Created report characterizing the IA domain based on review of research literature

• Developed adaptable method to identify 6 recommended interaction touch points for the MURI fusion system

•Challenges• Software state to conduct H‐I‐L experiments• Lack of direct access to Intelligence Analysts or SMEs

Main Scientific/Technical AccomplishmentsHuman Source Characterization & Human in the Fusion Loop

Characterized 8 additional soft observation categories – Based on existing empirical literature– Categories selected based on availability of literature + relevance to COIN IA

Created extensive report (+150 p.) characterizing the IA domain – Based on review of research literature– Focus on developing an understanding of the cognitive requirements,

complexities and constraints of the intelligence analysis domain

Developed methodology identify fusion system design components which support human operators in the fusion loop

– Based on a range of human factors engineering techniques– Method designed to be adaptable across fusion systems and design life‐cycle

stages– Utilized method to identified 6 recommended human‐fusion system interaction

touch points for the MURI fusion system2 of 21

Personnel supported:• Ph.D. Student: Michael Jenkins• Faculty: Ann Bisantz

Publications:• Journal papers: 1 (submitted)• Conference papers: 2 (published/presented)• Book chapters: 1 (in process)

Project Statistics and SummaryHuman Source Characterization & Human in the Fusion Loop

3 of 21

M. Jenkins, G. Gross, A. Bisantz, and R. Nagi. “Toward context‐aware hard/soft information fusion: Incorporating situationally qualified human observations into a fusion process for intelligence analysis,” IEEE Conference on Cognitive Methods in Situation Awareness and Decision Support, 2011, 22‐24 February, Miami Beach, FL. (8 pp).

M. Jenkins, G. Gross, A. Bisantz, R. Nagi “Towards Context Aware Data Fusion: Modeling and Integration of Situationally Qualified Human Observations into a Fusion Process for Intelligence Analysis,” IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, (Submitted, May 2011), 12 p.

M. Jenkins, A. Bisantz “Identification of Human‐Interaction Touch Points for Intelligence Analysis Information Fusion Systems,” Fusion 2011, 2011.Chicago, IL, July 5 ‐ 8.

M. Jenkins, A. Bisantz, J. Pfautz, (in preparation). Human Engineering Factors in Distributed and Net Centric Fusion Systems. In Distributed data fusion for net‐centric operations. Boca Raton: CRC Press.

PublicationsHuman Source Characterization & Human in the Fusion Loop

4 of 21

Overall Goal:Develop context‐aware error models for the remaining observation categories identified in year 1

– Only categories with existing empirical literature available to extract contextual influences & error characteristics were completed

– Approach was the same as was utilized for characterizing 4 categories completed in year 1

Categories Included:1. Memory based traversed distance estimation 2. Visual gender classification3. Visual quantitative estimation of object dimensions4. Memory based quantitative estimation of duration5. Visual quantitative estimation of weight6. Auditory based voice recognition (does not imply positive identification)7. Visual based facial recognition (does not imply positive identification)

Technical ApproachHuman Source Characterization

5 of 21

Overall Goal:Develop study to validate the benefits of providing context‐aware soft sensor source characterization models for information fusion systems

– No literature exists validating the fusion improvements when leveraging soft sensor source characterization models

– Experiment focused on comparing fusion output of simulated truthedand observed data with/without the context‐aware models

– Results to be analyzed using Signal Detection Theory

Potential Study Variables:1. Number of entity attributes2. Percentage of population with known attributes3. Fusion “Hit” threshold4. Comparison Format (Truthed vs. Observed & Observed vs. Observed)

Technical ApproachHuman Source Characterization

6 of 21

Overall Goal:Develop generalizable and adaptable method to support the design of fusion systems which support the human operator in the fusion loop.

Overall Approach:1. Characterize IA domain in terms of factors such as constraints,

actors, resources, dynamics, workflows, actors, etc.2. Apply HF and other techniques to the design of a theoretical fusion

system, based on predicted MURI project output, to:• Support human operator in the fusion loop• Enhance overall human‐machine system capabilities• Increase likelihood of successful system integration to existing IA workflows

3. Expand/Augment methodology to be adaptable to design of different fusion systems and across multiple domains

Technical ApproachHuman in the Fusion Loop

7 of 21

Intelligence analysis (IA) is a broad domain that in its most basic form is reasoning over available information with the goal of making a coherent whole of the past, present, and/or future states of some real‐world environment or situation.

– IA is a complex task of filtering, validating, associating and summarizing massive amounts of information relevant to a real‐world scenario

– IA is unlike most other analytical processes in that making a decision and taking action based on that decision is not the immediate goal

– IA outputs are passed to often remote consumers who will take action (and potentially face severe consequences) based on the provided summary

– IA is often considered to be unbounded as scenarios being studied must be considered on a dynamic and comprehensive scale, “the world is literally its province”

Intelligence analysts are challenged with this task of illuminating the future to provide “actionable intelligence,” by:

– Compiling, reviewing, and processing as much information as they can • Based on externally imposed deadlines and availability of data

– Providing an interpretation of some, often ambiguous, substantive problem • Based on their domain experience and hypotheses


Characterizing the IA Domain

8 of 21

Common Workflow:Needed to understand typical analyst workflow in order to define how, when & where a fusion system will aid analysts

• IA process is an iterative cycle; however 7 stages are typically present•Greater # iterations = more robust analysis

•Number of iterations is dependent on availability:• Of Time• Of New or updated information



9 of 21

Creation / Presentation of IA Artifact

Additional data collection & hypothesis testing

Moment of hypothesis

Evaluation of Data

Collection of data

Analysis of substantive problem

Appearance of substantive problem

Intelligence Analysis Complexities: • Leveraging an understanding of the IA domain & typical

workflow, common complexities were identified that presented challenges to analyst performance.

• Complexities were mapped to the stages of analysis based on when they are likely to occur. Likely Stage(s) of analysis where

Challenges will ariseCommon Challenges to IA Performance 1 2 3 4 5 6 7Ambiguous Requests for Information (RFI) XCharacteristics of the Data X X XDistributed Structure of IA Domain X XLack of Data Needed for Analysis X X XLack of Consumer and/or Source Feedback X XMultiple ‘Worlds’ of Consideration X X X XPotential for Deception X XDynamics of the Real‐World X XUnbounded Problem Space X X

Technical ApproachHuman in the Fusion LoopIA Complexities Mapping

Leveraging identified IA complexities, potential fusion system capabilities were identified to mitigate contributing factors.– Capabilities that focus on alleviating common IA complexities

are more likely to be leveraged by analysts than capabilities which simply automate a task that the analyst can perform manually with little effort or difficulties.


Identification of System Capabilities

IA Complexity 1 2 3

Ambiguous RFIs

Provide feedback channels to requesting source

Support multiple earches (stored or ad‐hoc) for situations of interest

Data CharacteristicsSupport manual or automated data association

Suppoautomassess

Distributed Structure

‐ Store access credentials to multiple data sources‐ Automate language translation

Provide ‐ Queue incoming data for Mainta

S

IA Complexity 1 2 3 4 5 6 7

Ambiguous RFIs


Support multiple earches (stored or ad‐hoc) for situations of interest


Data CharacteristicsSupport manual or automated data association

Support manual or automated situation assessment

Support manual or automated SA & DA

Distributed Structure

‐ Store access credentials to multiple data sources‐ Automate language translation

Lack of Data


‐ Queue incoming data for processing‐ Setup custom search alerts to notify when data or situation available appears

Maintain and fuse pedigree data during data association processes

‐ Support what‐if scenarios‐ Support automated pattern identification to highlight potential sit. of interest

‐ Queue incoming data for processing‐ Setup custom search alerts to notify when data or situation available appears‐ Maintain and fuse pedigree data during data association processes

Potential Deception

Auto‐validate credentials of request

Auto‐validate credentials of source

Support what‐if scenarios ‐ Support what‐if scenarios‐ Support automated pattern identification to highlight potential sit. of interest

‐ Support what‐if scenarios‐ Support automated pattern identification to highlight potential sit. of interest

Real‐World Dynamics

Maintain temporal meta‐data of requests

‐ Update KdBase in near real‐time with most recent data‐ Create temporal boundaries on SA & DA‐ Maintain temporal reference meta‐data

‐ Update KdBase in near real‐time with most recent data‐ Create temporal boundaries on SA & DA‐ Maintain temporal reference meta‐data

Unbounded Problem SpaceSupport custom boundary ranges

Support custom boundary ranges

Support custom boundary ranges

Stage of Intelligence Analysis Process

The capabilities mapping table is used to:• Track the development of each capability• Ensure the final fusion system provides coverage to mitigate all the common complexities• Highlight areas where capabilities are needed (pink cells with no text)• Highlight areas where potentially excessive capabilities are being included (grey cells with

text)


Complexities:Fusion Sys. Capabilities Mapping

12 of 21

Human‐System touch points were identified to facilitate the incorporation of the identified system capabilities into the larger fusion architecture.– Touch points selected given considerations of user, system and

workflow/domain characteristics


Identification of Recommended Touch Points

User Requirements &

Limitations

System Requirements & Limitations

Domain / Workflow Dynamics & Constraints

Overlapping regions between User & System are where potential touch points would be required.

13 of 21

Touch Point

Touch Point

Touch Point

Touch Point

Touch Point

Touch Point


Architecture Touch Point Locations

14 of 21

Individual touch point features were identified to begin defining an effective & efficient human‐computer interaction for each respective touch point.

• Features selected to instantiate the pre‐defined capabilities

Touch point definitions were limited by the early stage of the fusion system architecture.

• User interface mediums were not defined• System input/output formats were not defined• Availability of source/data meta‐information not defined

As the system backend and architecture become better defined, further effort will be dedicated to defining the touch points

• Focused on the human‐computer interaction


Defining Fusion Sys. Touch Points

15 of 21

Location: After data association processing, but prior to incorporation of processed data into the overall fusion entity‐association database

Expected IA Stage(s) When Accessed:Stage 3 & 4 – Collection & Evaluation of Data

Complexities being at least partially addressed:• Distributed Structure of IA Domain• Potential for Deception• Real‐World Dynamics

Required Features to Allow For:• Browsing/reviewing of incoming data sets’ entity‐association network• Selection of entities within the network to highlight or annotate prior to

fusing with the overall fusion database• Review of executed data merges• Search input to search the incoming data sets’ network for situations of

interest• Drill‐down to entity/association meta‐data, source data, and data

association log

Example Touch Point



Touch Point

16 of 21

Technical ApproachHuman in the Fusion LoopModel Generalization

Methodology was adapted to provide:– Generalized methods applicable to alternative domain & system architectures– HF engineering techniques to apply at different stages of system developmentEarly Development

Stage

1: Characterize Domain

2: ID Potential Sys Features to Overcome Domain Challenges

3: Characterize Intended User

Mid‐to‐Late Development Stage

4: Designate Touch Points to Control,

Monitor, Access, etc. Sys Capabilities

5: Apply HCI Methods to Design of Touch Point Dialogues

6: Plan/Conduct human‐in‐the‐loop XPs to Validate Touch

Point Selections

Post‐Release Stage

7: User Training

17 of 21

1. Generated context‐aware error characteristics for 8 additional soft observation categories

2. Created extensive report characterizing the IA domain based on review of research literature

3. Developed adaptable methodology to apply human factors engineering to the design and/or improvement of fusion systems across life‐cycle stages– Applied method to identify & initially define requirements for 6 human‐

fusion system interaction touch points for the MURI fusion system

Challenges: • MURI system maturity not yet supportive of high fidelity HIL studies• Lack of direct access to Intelligence Analysts (or other SMEs)

SummaryHuman in the Fusion Loop

18 of 21

Capability Goal: Context‐Aware Human Source CharacterizationsTo validate the benefits of providing soft sensor context‐aware error characteristics for fusion processes, an experiment will be designed and conducted leveraging a subset of the 12 observation category error models.

Research Goals: Validate Human Source Characterization EffortsDetermine the degree of fusion system improvement gained from leveraging the source characterization models for both:– Truthed to Observed References & Associations– Observed to Observed References & Associations

2011‐2012 PlansHuman Source Characterization

19 of 21

2011‐2012 PlansHuman in the Fusion Loop

20 of 21

Capability Goal: Human‐in‐the‐LoopGiven that adequate system components are unlikely to exist for model prototyping, plan and conduct human‐in‐the‐loop experiments focused on topics generalizable to MURI fusion system planned capabilities and employment

Research Goals: Concept‐of‐EmploymentTouch Point 5: •Determine effectiveness of input formats for operators querying fusion systems

Touch Point 6:•Determine effectiveness of output formats for operators leveraging fusion system• Explore the benefits (& drawbacks) of integrating level 2 fusion systems into IA workflows

Touch Point 5

Touch Point 6

Capability Goal: Human Operator SupportHard‐soft fusion system should be supportive of multiple forms of human operator interaction as identified by methodology employed in 2010‐11 program year.

Research Goals: Human‐in‐the Loop experimentation• Leverage results of experimental & theoretical studies to support design of human touch

points within MURI system under development to deploy:• Touch Point 3: Validated control actions to support user during data association processes

• Touch Point 4: Validated control actions to support user during graph merging and enhancement operations

• Extend experimental methodology to encompass multiple touch points as well as to use the capabilities of the MURI systems as those capabilities become available

2012 – 2014 PlansHuman in the Fusion Loop

21 of 21

Touch Point 3

Touch Point 4

APPENDIX

Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.

• Intelligence analysts receive problem of focus most often as direct instance of a consumer request (e.g., Tell me about this)

•Problems, or sub‐problems, can also arise as result of:• Emergence of something unusual (e.g., What is that?)

• An effort to anticipate future problems (e.g., What if this were the case?)

• Source of the problem often affects the complexity & degree of difficulty of the subsequent analysis.

Stage 1: Appearance of substantive problem



23 of


• This stage is based on framing & understanding the problem to serve as a starting point for evidence gathering & evaluation.

• The goal here is to reduce the ambiguity of the problem facing the analyst

• The degree of difficulty is often based on type of request & the degree of ambiguity:•Overly general requests are more common in terms of challenging requests (e.g., Tell me everything about XYZ?)

•Overly narrow requests can also represent a potential challenge (e.g., Where is person A right now?)

Stage 2: Analysis of substantive problem



24 of 26

• Rounding up ALL available materials the analyst believes may be related to the problem environment.

• Depending on the availability of information this can result in the potential for:

• Data overload • Challenging scenarios where there is a large gap in available information.

• Due to distributed nature of IA, problems can arise if analysts need data:

• From another organization• From another expert• On a domain he/she is not familiar with

• Across a language barrier

• Collaboration can further add to the complexity of the issue due to the analyst needing to quickly and effectively convey the context of his/her request(s).

Stage 3: The collection of data



25 of 26


• The task of reading and understanding the collected information to determine how it compares to the analyst’s hypotheses and previously considered information.

•A “criticism of data,” as it should consist of the analyst questioning the information’s: •Pedigree •Relevance in relation to all other available data

Stage 4: The evaluation of data



26 of 26


• This is a stage or moment where the analyst puts together the pieces to form the inklings of an explanation for the evaluated data.

• Ideally there will be numerous hypotheses that the analyst considers in parallel•Due to limitations of human cognitive capabilities ideal comparison set is rarely considered

Stage 5: The moment of hypothesis



27 of 26


• Iteration of the previous stages based on analyst’s current hypotheses & the marshaling of evidence to confirm and/or refute them.

• This stage of the process is often dependent on the availability of:• Time •New or updated information

• Evaluation of hypotheses shown to be susceptible to common issues due to human cognitive capabilities. Ex:

• Confirmation bias• Tunnel vision

Stage 6: More collecting/testing of hypotheses



28 of 26


• Final stage of the analyst’s process is creation of an artifact to communicate the established hypothesis (or ideally competing hypotheses) to the IA consumer(s)

•Analysts are not decision makers, their job is to develop & successfully transfer a “new and better approximation of the truth” with regards to the situation of interest

•Distributed nature of domain often makes consumer feedback inaccessible so analysts do not know if their analysis was on target & actionable

Stage 7: Creation / Presentation of IA Artifact



29 of 26

Location: • Prior to incoming data set processing

Expected IA Stage(s) When Accessed:• Stage 2 – Analysis of Substantive Problem

Complexities being at least partially addressed:• Ambiguous requests for information• Lack of Data

Required Features:• Allow for custom data sets to be created• Allow for multiple data sets to be prioritized for

processing• Requires ability to scan/review data sets pre‐processing

Touch Point 1:



Location: • After initial processing & uncertainty alignment, but prior to data

association

Expected IA Stage(s) When Accessed:• Stage 3 – Collection of Data

Capability Intended to Support:• Support multiple searches (stored or ad‐hoc) for situations of

interest

Required Features to Allow For:• Review of data sets being considered for data association• Filtering of data sets to determine custom boundaries to utilize for

data association• Selection of additional data sets (previously processed) to be

included in data association• Custom threshold levels to be set to determine when manual

approval is needed for a system proposed merge to be carried out• Custom uncertainty values to be assigned to individual data

elements (e.g., what if scenarios)

Touch Point 2:



Location: • During data association processing

Expected IA Stage(s) When Accessed:• Stage 3 – Collection of Data

Complexities being at least partially addressed:• Data Characteristics• Potential for Deception

Required Features to Allow For:• Review of system proposed merges based on pre‐determined threshold level

• Approval/Rejection assignment to system proposed merges

Touch Point 3:



Location: • After data association processing, but prior to incorporation of processed

data into the overall fusion entity‐association database

Expected IA Stage(s) When Accessed:• Stage 3 & 4 – Collection & Evaluation of Data

Complexities being at least partially addressed:• Distributed Structure of IA Domain• Potential for Deception• Real‐World Dynamics

Required Features to Allow For:• Browsing/reviewing of incoming data sets’ entity‐association network• Selection of entities within the network to highlight or annotate prior to

fusing with the overall fusion database• Review of executed data merges• Search input to search the incoming data sets’ network for situations of

interest• Drill‐down to entity or association meta‐data, source data, and data

association log

Touch Point 4:



Location: • After any update to the fusion system information database

Expected IA Stage(s) When Accessed:• Stage 3 & 4 – Collection & Evaluation of Data

Complexities being at least partially addressed:• Distributed Structure of IA Domain• Potential for Deception• Real‐World Dynamics• Unbounded Problem Space

Required Features to Allow For:• Browsing/reviewing of the fusion database• Selection of database boundaries with respect to search/browsing

capabilities• Review of highlighted/annotated entities and/or associations• Manual editing/addition of entities/associations/attributes• Search input to search the database using custom criteria• Creation of entity/association placeholders that indicate expected

hypotheses not yet incorporated/observed• Drill‐down to entity or association meta‐data, source data, update log,

weighting, edit precedence, data association log

Touch Point 5:



Location: • After any update to the fusion system information database

Expected IA Stage(s) When Accessed:• Stage 6 – Additional data collection & hypothesis testing

Complexities being at least partially addressed:• All, with the exception of ambiguous requests for information

Required Features to Allow For:• Review of data sets being considered for data association• Filtering of data sets to determine custom boundaries to utilize for

data association• Selection of additional data sets (previously processed) to be

included in data association• Custom threshold levels to be set to determine when manual

approval is needed for a system proposed merge to be carried out• Custom uncertainty values to be assigned to individual data

elements (e.g., what if scenarios)

Touch Point 6:



Documents

Bisantz and Michael Jenkins, University at Buffalonagi/MURI/MURI/Year_2_files... · 2011. 9. 26. · Human Source Characterization & Human in the Fusion Loop Ann Bisantz and Michael