T OPIC 16: S TATE OF THE A RT David L. Hall. T OPIC O BJECTIVES Provide an overview of key technology trends associated with data and information fusion

TOPIC 16: STATE OF THE ART

David L. Hall

TOPIC OBJECTIVES

Provide an overview of key technology trends associated with data and information fusion

Discuss predictions and implications for future system development

THESIS

The rapid evolution of Information technology (IT) provides major opportunities for the information fusion to improve the ability to accomplish a new generation of applications and missions;

These same technologies also provide major threats to the ability to accomplish these missions

A major challenge for the sensing and processing community is how to take advantage of IT opportunities for improved mission effectiveness,

while minimizing threats posed by these very same technologies

TECHNOLOGY TRENDS: SENSORS

TRENDS IN COMPUTING

http://www.en.wikipedia.org/wiki/Moore’s_law

• Stovepipe systems

• Little or no interoperability

• Some network connections

Pre-Web. . .. . . Joint / Enterprise

• Pervasive networksPervasive networks• Mission-effective apps & applets Mission-effective apps & applets • Assured, interoperable enterprise services Assured, interoperable enterprise services • Dynamically composable architecturesDynamically composable architectures• Robust & reliable edge computing Robust & reliable edge computing • Accurate, timely & relevant infoAccurate, timely & relevant info• Improved Quality of Service (QOS) with Improved Quality of Service (QOS) with

centrally managed infrastructurecentrally managed infrastructure

. . .Today. . .

• More networks• Some web services• Various directory &

security services• Uncoordinated

Service/Functional transformations

• Few authoritative data sources

EnterpriseServices

ForceSustainment

Providers

This slide is from a DISA briefing by Rob Walker, April 20, 2004

TRENDS IN CONNECTIVITY: THE ROAD TO NETWORK CENTRIC

WARFARE

OPPORTUNITIES AND OPPORTUNITIES AND CHALLENGESCHALLENGES

Information Technologies Data (ubiquitous and persistent surveillance) Mobile computing Increasing network speeds & connectivity Cloud computing Advanced multi-sensory human computer

interaction

Information The exploding digital universe Meta-data generation Hard and soft data fusion

People Digital natives and net-generation Participatory sensing

DISTRIBUTED SENSOR NETWORKS MILITARY APPLICATIONS

Source: www.plansys.com - Sensor Networks for Network-Centric Warfare

Source: Daniel Van Hook, MIT, “Dynamic Declarative Networking”, DARPA SensIT meeting, Oct 7-8, 1999

• Battlefield operations (detecting, locating, tracking, and identifying targets)

• Situation and context awareness (humans and computational devices)

• Machinery performance and malfunction monitoring

WHAT WE WANT: EMPOWERED USERS

Graphic, Applied Computing 4-Day Conference, www.ac-

conference.com, 2000.

InformatiInformation on

FusionFusion

instrumented, networked

world

The knowledge empowered analyst

THE X-MEN DATA INTERFACE

THE ULTIMATE LIMITING RESOURCE IS HUMAN

ATTENTION UNITS (HAUS)

TRANSFORMATION FROM ENERGY TO KNOWLEDGE

Energy Signals Data State vectors Labels Knowledge

The utility of a data fusion system must be measured by the extent to which it supports effective decision making

The longest yard

THE JDL DATA FUSION PROCESS MODEL

HumanComputerInteraction

Sources

DATA FUSION DOMAIN

Level FourProcess

Refinement

SourcePre-Processing

Level One:Object

Refinement

Level Two:Situation

Refinement

Database Management System

SupportDatabase

FusionDatabase

Level Three:Threat

Refinement

LevelFive:

Cognitive Refinement

THE DOD LEGACY: EXTENSIVE RESEARCH

INVESTMENTS• JDL Process model• Taxonomy of

Algorithms• Lexicon• Engineering Guidelines

– Architecture Selection– Algorithm Selection

• Evolving Tool-kits• Extensive Legacy of

technical papers, books

• Training Materials• Test-beds• Numerous prototypes

JDL LEVEL 0: PRE-PROCESSING

Recent Advances

• Ubiquitous nano-scale, smart, low-cost sensors

• Embedded signal & image processing• Increased agility in tasking & sensor

operation• Improved modeling of sensor

performance and signal propagation

Advanced sensors and improved processing at the sensor level provide the opportunity for ubiquitous sensing and accurate modeling of sensor performance

Challenges and opportunities

• Establishment and tracking of pedigree• Common representation of sensor

performance• Web-based plug & play service approach• Human carried sensors• Web-service advanced models for sensor

performance & propagation• Counter-measures & information warfare

JDL LEVEL 1: TRACKING

Recent Advances

• Extensions of classic filtering & estimation

• Advanced methods in multiple hypothesis tracking

• Continued evolution of random set theoretic methods

• Incorporation of identify information to resolve correlation ambiguities

• Distributed processing implementations

Target tracking is becoming increasingly mature, evolving towards increased accuracy and sophistication


• Increasing sophistication in move-stop-hide strategies

• Relatively rapid movement of targets• Tracking of individual humans• Generalized concepts of target & entity• New algorithms for addressing correlation

& interdependent sensor measurements

LEVEL 1 JDL PROCESSINGRecent Advances

• Advances in implicit pattern recognition techniques such as machine learning methods and hybrid implicit/explicit methods

• High performance computing allows physics based modeling of observable features

• Hybrid methods incorporate both implicit and explicit information

• Automated generation of semantic “meta-data” for images and signals (e.g., semantic labeling of images)

Object identification transforms observed attribute data into a label or declaration of target identity


• Increased sophistication in multi-Intelligence spoofing

• Relatively rapid movement of targets• New user-centric approaches to object

identification and information aggregation• Improved observation prediction using

high-performance target models

LEVEL 2 AND 3 JDL PROCESSING

Recent Advances

• Utilization of agent-based technologies• Fuzzy logic reasoning• Hybrid implicit/explicit reasoning

techniques• Incorporation of semantic & numerical

information• Maturation of Bayes & D-S Belief Nets

Level-2 & Level-3 fusion is very challenging; it involves the attempt to emulate human reasoning


• What is a target now anyway?• Emerging techniques for image

representation via semantic concepts• Leveraging of new search engines• Advances in cognitive modeling• Collaboration between automated

(machine) processing and humans-in-the-loop

• Distributed collaboration “crowd-sourcing”

JDL LEVEL 4 PROCESSING: PROCESS REFINEMENT

Recent Advances

• Ubiquitous nano-scale, smart low-cost sensors

• Wide-bandwidth communications• Web-services approach for “plug-in”

services• Emerging community standards related

to information representation• Advances in control theory

Ubiquitous sensing, wide bandwidth communications & distributed processing provide both opportunities & challenges for sensor and process control & optimization


• E-business models for auction-based resource optimization

• Multi-time scale sensors & processing requirements

• Human-in-the-loop optimization• Utilization of agent-based methods • Need for multi-source validation

JDL LEVEL 5 PROCESSING: COGNITIVE REFINEMENT

Recent Advances

• 3-D displays• Haptic interfaces• Gesture recognition & Natural Language

Processing (NLP)• Computer aided cognition &

collaboration• Agent models of team interaction

New HCI technology and cognitive models provide opportunities for enhanced effectiveness of the human-fusion system


• Overwhelming data from “live” feeds• Cognitive limitations and biases• Collaboration from multiple experts

(different expertise, cultures, decision styles)

• Rapid technology advances from the e-world (affective computing, adaptive interfaces)

CROSSING THE LONGEST YARD

Whole brain analysis – automated semantic labeling of images

User-centered data correlation and fusion Cognitive aids for bias remediation Adaptive search engines – dynamic adaptation to users

and content of searched material Hybrid (implicit and explicit) reasoning Team-based advisory agents Multi-expert collaboration aids Advanced visualization tools

WHOLE BRAIN ANALYSIS CONCEPT

Support Analyst Support Analyst

SituationAssessment

SituationAssessment

Text-based- Search engines- Semantic nets- Template

Processed Text

Raw text

Tex

t to

ima

ges/

disp

lays

Imag

es

to te

xt

Collection

Text Processing

Interpretation

• Open Sources• HUMINT• Models

Visualization of non-physical phenomena

Automated annotation of images with semantic labels

Interpretation

Image Processing

Sensor

Raw data/images

Processed Images

Image-based- Visual displays- Interpretation techniques- Overlays & comparisons- Visual reasoning tools

SituationAssessment

SituationAssessment

ADVANCED DISPLAYS/HCI

Improved Data Understanding Improved information

understanding Information discovery Multi-expert collaboration Utilization of visual intelligence for

data mining and understanding Utilization in an operational

environment Development of tools for effective

collaboration

3-D Immersive Display

Hierarchical Layered 3-D Display

VISUALIZATION AIDS

• Located events over time, with time as months on the z axis

• Icon shape denotes target• Icon color designates

perpetrator• Label describes attack type• DTED textured with TPC

map• Navigation through

environment• Query event for additional

information

Event Pattern Visualization

DELIBERATE SYNESTHESIA

Synesthesia• Stimulus in one sense modality involuntarily elicits a

sensation/experience in another sense modality• Involves making arbitrary links between seemingly unrelated

perceptual entities such as colors and numbers• Cross-modal abstraction in which hearing and vision flow

together to enable the construction of high-level perceptions

INTELLIGENT AGENTS FOR DECISION SUPPORT

Shared MentalModel

Information Fusion 2+

Information Fusion 1



Team DecisionContext

Computational SMMContext

Shared MentalModel













GAMING APPROACHES FOR DATA UNDERSTANDING

Knowledge Extraction (Single point

in time)


in time)


in time)

Visual Analysis

Text Analysis


in time)

Pattern Extraction

Storification

Linking Events in time


in time)


in time)


in time)

Visual Analysis

Text Analysis


in time)

Pattern Extraction

Storification

Linking Events in time

Forensic Photographer

Bidding for Data

Ghost Story Endangered Species

OBJECTIVEDemonstrate a mainstream gaming approach to rapidly model and evaluate the high-level design of space-based global reconnaissance systems against real-world scenarios

RESULTSGaming Test Bed

• Player interface, simulation, scoring, and metrics– Collection simulation – Cost allocation and budget– Game theoretic 3-player extensive form scoring- Traditional/non-traditional threat scenarios

• Diverse, multi-disciplinary experiment trial participation• Enabled players to rapidly create successful architectures

– 82% of players improved performance– Game environment stimulated/motivated participants

EXPERIMENT TRIALS• Group 1: Gaming experience,

no IC domain• Group 2: IC domain experience• Overall, Group 1 outperformed

Group 2

RAPIDLY MODELING WINNING: GLOBAL RECONNAISSANCE

SYSTEMS

AN INTEGRATED EXAMPLE

Analyst Support Functions• Search of images using semantic labels• Find “like” images• Context-based interpretation of images• Multi-analyst collaboration aids• Anomaly/event detection• Analyst/information interface tools

S1

S2

SN

• •

HUMINTOpen-Source

Automated Semantic Labeling

Information Sphere• Selected images• Semantic labels• Interpretation layers• Context spatial information

Semantic Labels

Text-based & parametric

searches

Context-based Reasoning

Multi-modal interaction

DIRTY SECRETS IN DATA FUSION REVISITED

There is still no substitute for a good sensor (and a good human to interpret the results)

Downstream processing still cannot absolve upstream sins No only may the fused result be worse than the best sensor – but failure

to address pedigree, information overload, and uncertainty may really screw up things

Still no magic algorithms (yes, even agents, ontologies, D-S nets, etc.) Never enough data (but we can help by hybrid methods) We’ve started at the wrong end (and continue to focus on the wrong end) It’s not the data!!!

TOPIC 16 ASSIGNMENTS

Preview on-line materials for topic 16 and topic 17 Read Hall and Steinberg chapter referenced above Team assignment (T-6): Complete and submit your

final report describing the data fusion system design for your selected application

DATA FUSION TIP OF THE WEEK

He who gazes into the crystal ball must be prepared to eat ground glass!

Forecasts of the future are tainted by:

- Zeitgeist effects

- Failure to understand economics

- Unanticipated consequences

- Lack of imagination

Documents

T OPIC 16: S TATE OF THE A RT David L. Hall. T OPIC O BJECTIVES Provide an overview of key technology trends associated with data and information fusion