Revisiting the JDL Data Fusion Revisiting the JDL Data Fusion Model IIModel II

James Llinasa, Christopher Bowmanb, Galina Rogovac, Alan Steinbergd, Ed Waltze , and Frank

Whitef

a: Research Professor, University at Buffalo. Buffalo, NY, USA, llinas@eng.buffalo.edub: Consultant, Data Fusion & Neural Networks, Colorado, USA, cbowman@indra.com c: Encompass Consulting, Honeoye Falls, NY, USA, rogova@rochester.rr.comd: Technical Director, Utah State University Space Dynamics Lab, Utah, USA, Alan.Steinberg@sdl.usu.edue: Technical Director, Intelligence Programs, General Dynamics - Advanced Information Systems, Ann Arbor, MI, USA, ed.waltz@gd-ais.comf: Director, Program Development, US Navy SPAWAR Systems Center, San Diego, CA, USA, frank.white@navy.mil

Some History of Fusion ModelsSome History of Fusion Models

• JDL—Original, circa 1987• Dasarathy, Data-Feature-Decision Layered

Model—1997• Steinberg, Bowman, and White, Revision I to

JDL, 1999• Bedworth and O’Brien, Omnibus Model,

1999-2000• Salerno, Situation Awareness Model, 2002• Blasch and Plano, Level 5, 2003

The Reference JDL Model*The Reference JDL Model*

* Steinberg, A.N., Bowman, C.L., and White, F.E., “Revisions to the JDL Data Fusion Model”, in Sensor Fusion: Architectures, Algorithms, and Applications, Proceedings of the SPIE, Vol. 3719, 1999

Motivations for Revisiting the JDL Model(s)Motivations for Revisiting the JDL Model(s)

• “Common (or Consistent, or Relevant or Single Integrated or User Defined) Operational Picture”

• “Network-Centric Warfare”• “Dominant Battlespace Knowledge”• “Operations Other Than War”• “Asymmetric Warfare”• “Information Warfare”• “FORCEnet”

Distributed, Service-BasedInformation Architectures

Dynamically-Composable Data and Information

Fusion Services

Pedigree, Metadata, Context Services Conventions and Standards

Ontologies

Underlying implications for the primary conceptual and semantic DF model:

The JDL Model

“External” Factors(Driven by Opnl Needs)

“Internal” Factors(Driven by Need for

Deeper Understanding)

Better understanding of the “Levels”

Insight into Inter-Level Processing--Information operations

--Adjudication and conflict resolution--Output management

--Effects of Input Reliability

Integrating Inductive/Abductive Inferencing

Discussion TopicsDiscussion Topics

1) Reexamining our understanding of the “Levels”

2) Insight into Inter-Level Processing--Information operations

--Adjudication and conflict resolution--Output management

--Effects of Input Reliability

3) Integrating Inductive/Abductive Inferencing

4) Aspects of Distributed Fusion

5) Pedigree

6) Ontologically-based Data Fusion Processes

1) Revisiting the “Levels”1) Revisiting the “Levels”

• For Alan to do—some bullets on new perspectives re Levels

1) Revisiting the “Levels” cont’d 1) Revisiting the “Levels” cont’d

Evaluation (Situation to

Actor’s Goals)

Assignment(Observation-

to-Entity)

[Action][Control][Planning

(Resource to Task)]

L.4 – Process Refinement

EstimatedSituation Utility

Game-TheoreticInteraction

L.3 – Impact Assessment

Estimated Situation State

RelationRelationship

(Entity-to-Entity)

L.2 – Situation Assessment

EstimatedEntity State

Attributive State

L.1 – Object Assessment

EstimatedSignal State

DetectionAssignment

(Observation-to-Feature)

L.0 – Signal Assessment

ProductEstimation Process

Association Process

Data Fusion Level

2) Insight into Inter-Level Processing2) Insight into Inter-Level Processing(a) Information operations(a) Information operations

• The idea of inter-Level information and control flow is not very explicit in the traditional JDL Model

• Need to specify inter-Level “informing”, controlling, and exploitation • Trades off added value/utility vs cost of additional processing; raises need

for consistency

L ev el " m " L ev el " n "S o m e p re -d e s ig n e d alg o rith m ic

m e an s to e x p lo it in L e v e l m in fe re n c in g

S o m e co n tro lk n o wle dg e th a t a s s e rt sth a t it is " v a lu a ble " to

in fo rm L e v e l n o fs o m e th in g th a t L e v e l m

k n o ws a bo u t , e g" - e n h a n ce d in fe re n ce s "

I n te r- le v e lA dju dica t io n : I s

co n s is te n t witho n g o in g L e v e l m

in fe re n ce s ?

I n fo rmL e v e l mo f ""

S o m e co n tro lk n o wle dg e th a t a s s e rt sth a t it is " v a lu a ble " to

in fo rm L e v e l m o fs o m e th in g th a t L e v e l n

k n o ws a bo u t , e g""

Data Fusion Tree NodeData Fusion Tree Node

Prior Data FusionNodes & Sources

Data Preparation(Common

Referencing)

HypothesisGeneration

HypothesisEvaluation

HypothesisSelection

StateEstimation

&Prediction

DataCorrelation

Data Fusion Tree Node

User orNext Fusion

Node

• Detect and resolve data conflicts• Convert data to common time and coordinate frame• Compensate for source misalignments

• Gating and generation of feasible and confirmed association hypothesei• Scoring ofhypothesized data associations

• Select, delete, or feedback data associations

• Estimate/predict object& aggregate states - Kinematics. attributes, ID - From each perspective (blue, red)•Estimate sensor/source misalignments•Feed forward source/sensor status

Source/Sensor Status Resource Management Controls

Data Association

Operations Within/Across LevelsOperations Within/Across Levels

State Estimation Bias

Level 0 Level 1 Level 2 Level 3

Level 4

Inter-Level Considerations

2) Insight into Inter-Level Processing2) Insight into Inter-Level Processing(b) Adjudication and conflict resolution(b) Adjudication and conflict resolution

• Both Atomic Level and Meta-Level Adjudication

L e ve l nR e fi ne m e nt

L e ve l mR e fi ne m e nt

N o dalP r o c e s s i ng

as pe r F i g . 2

N o dal P r o c e s s i ngas pe r F i g . 2

D ataM anag e r

L e v e l-s p e c ificD ata

IN P U T S

L e v e l-s p e c ificD ata

L e ve l n"Info r m i ng " Info r m ati o n

Adjudi c at i o nN o t R e s o l ut i o n

L e ve l mF us i o n / E xpl o i tat i o n

Val ue Adde d

Thr e s ho l d

Sto p / N o t

I n te r-L e v e l P ro c e s s in g

2) Insight into Inter-Level Processing2) Insight into Inter-Level Processing(c) Output management(c) Output management

• JDL Model not specific in how output Quality & Consistency are controlled

• Expect hierarchical Value system; within-process and system-level

W ith in -N o d e O p e ra tio n s

L e ve l 3L e ve l 1 L e ve l 2

W ith in -L e v e l & I n te r-L e v e lV alu e -ad d in g S e rv ic e R e q u e s ts

L e ve l 0

N e wS tate E s tim ate

L e v e l 4P ro c e s s

R e fin e m e n tS e rv ic e R e q u e s tA rb itratio n an d

S e rv ic in g

C u rre n tS tate E s tim ate

S y s te m / No da l O u tpu t- -Q u a lity ch e ck e d

- -C o n s is t e n cy ch e ck e d

S y s te m / No da lO u tpu t C o n tro l

Q u a lity C o n tro l- - S er v ic e R eq u es ts

C o n s is t e n cy C o n tro l- - Belie f C h an g e

Output Inferencing Quality Control via Addtl info using L4

Output Inferencing Consistency via Belief Change

New State Estimate Quality andConsistency achieved as per

operations in 2(a),2(b)

2) Insight into Inter-Level Processing2) Insight into Inter-Level Processing(d) Effects of Input Reliability(d) Effects of Input Reliability

• Reliability akin to second-order Uncertainty (in source inputs)

• Typically not accounted for in fusion algorithms

• Even if Source Reliability specified, how to compute Fused-Estimate Reliability?

Possible situations (Dubois and Prade, 1992) • It is possible to assign a numerical degree of reliability to each source. • A subset of sources is reliable but we do not know which one.• Reliabilities of the sources can be ordered but no precise reliability values are known.

Possible situations (Dubois and Prade, 1992) • It is possible to assign a numerical degree of reliability to each source. • A subset of sources is reliable but we do not know which one.• Reliabilities of the sources can be ordered but no precise reliability values are known.

Strategies to be considered:•Strategies for identifying the quality of data input to fusion processes and elimination of data of poor reliability. •Strategies for modifying the data and information by considering their reliability before fusion. •Strategies for modifying the fusion process to account for the reliability of the input. •Combination of strategies mentioned above.

),...,(),...,( 11 IRI xxFxxF

RF - is a context dependent operator, which depends on the strategy selected and is defined within the framework used for uncertainty representation

3) Integrating Inductive, Abductive Inferencing

• Asymmetric adversaries are quite unpredictable in their behavior, tactics, weapons, and choice of targets.

• Induction usually a precursor to Deduction but requires knowledge of relationship between observable signatures and Truth states

• Abduction forms best plausible explanation for the observables and observable patterns

• A Hybrid Inferencing Process follows the typical sequence of scientific discovery and proof, using a sequence of steps to conjecture, hypothesize, generalize and validate.

Discovery

• Data mining tools to locate patterns of meaningful relationships • Correlated patterns are examined for relevance • Abductive Phase

Generalization & Validation

• Applies inductive generalization• Model parameters are estimated

Detection

• Validated model provides a target

detection “template’

3) Integrating Inductive, Abductive Inferencing Integrated Data Mining and Data Fusion ProcessesIntegrated Data Mining and Data Fusion Processes

Step Process Reasoning Process Example use of Typical Automated Tools

1.

DiscoveryData Mining – Discovery of

a potential specific target and it’s characteristics in raw data sets

Abduction – Reason about a specific target, conjecturing and hypothesizing to discover the best explanation of relationships to describe a target. (Hypothesis creation)

Analyst uses data mining tools to locate patterns of relationships in contacts, financial exchanges, associates, and concurrent activities of a terrorist cell.

2.

Generalization and Validation

Target Modeling Generalization – Characterize the target class in a general model

Induction – Generalize the fundamental characteristics of the target in a descriptive model. Test and validate the characteristics on multiple cases. (Hypothesis validation)

Analyst develops sand refines a quantitative model of the terrorist cell behavior. The model is tested on additional data to evaluate its detection value using data mining Tools.

3.

DetectionData Fusion – Detection of

subsequent occurrences of the target based on comparison with target models.

Deduction – Test real-time and massive volume data against multiple target templates to detect (deduce) the presence of targets. (Hypothesis testing)

Real time raw data are ingested by an automated data fusion tool to detect the presence of evidence for other similar terrorist cells.

3) Integrating Inductive/Abductive Inferencing *

1

DataWare-house

Data Fusion

Real-time detectionof known patterns

Data MiningOff-line discovery

of new patterns

2

3

Source 1

Source 2

Source 3

Le

ve

l 0

Le

ve

l 1 F

usio

n

Le

ve

l 2 F

usio

n

OperationalData Stores

ObjectDatabase

Le

ve

l 3

SituationDatabase

Da

ta C

lea

nsin

g

Tra

nsfo

rm

EntityRelationship

Database

Entity-RelationshipClustering

Mo

de

lC

reatin

g

Vis

ualiz

atio

n

Situations

Impacts

ExtractTransform

Load

An

aly

ticG

ene

raliz

atio

n-V

alid

atio

n

Mo

de

lT

estin

g

Discovered templates

1

DataWare-house

Data Fusion

Real-time detectionof known patterns

Data MiningOff-line discovery

of new patterns

2

3

Source 1

Source 2

Source 3

Le

ve

l 0

Le

ve

l 1 F

usio

n

Le

ve

l 2 F

usio

n

OperationalData Stores

ObjectDatabase

Le

ve

l 3

SituationDatabase

Da

ta C

lea

nsin

g

Tra

nsfo

rm

EntityRelationship

Database

Entity-RelationshipClustering

Mo

de

lC

reatin

g

Vis

ualiz

atio

n

Situations

Impacts

ExtractTransform

Load

An

aly

ticG

ene

raliz

atio

n-V

alid

atio

n

Mo

de

lT

estin

g

Discovered templates

* Waltz, Edward L., “Information Understanding: Integrating Data Fusion and Data Mining Processes”, Proc. of IEEE International Symposium on Circuits and Systems, Monterey CA, May 31-June 4, 1997. For a more detailed description of the integration, see, Waltz, Edward, Knowledge Management in the Intelligence Enterprise, Norwood MA: Artech, 2003, Chapter 8.

4) Aspects of Distributed Fusion

• Requirement, framework for sensibly all modern, future military, homeland security IT environments– Architectural issues—need for empirical studies,

architectural analysis tools• Need for local and network fusion algorithms

• Specification of Information-Sharing Strategies

• Design of adaptive network topologies

– Need for a “Distributed Fusion JDL Model”

Extensions to the Distributed CaseExtensions to the Distributed Case

L e v e l 1 L e v e l 3L e v e l 2

Async h,P ar al l e l

butC o ns i s te nt

Ac r o s s L e ve l s

L e v e ls 0 , 1 L e v e l 3

L e v e l 4

L e v e l 2 L e v e l 5

I n fo rm s

I n fo rm s

Le ads to :-Expl o i ta ti on

--Fu s i onwi th i n th i s Le ve l

Le ads to :-Expl o i ta ti on

--Fu s i onwi th i n th i s Le ve l

N o dal Sys te m O utput

O rgan i cM s m ts

Ne two rk Pro ce s s e s

R e c e i vi ng N o de Adju di ca ti onProce s s i n g

Se ndi ng N o de N e tworkEs ti m ate s

Pe di gre eIn form ati on

Pe dig re eI n fo rm a t io n

N e tworkM s m ts

In form ati on-S h ari n gS trate gy

Pe di gre eIn form ati on

W ith in -N o d e

N e twork(In te r-N ode )

Ne two rk M s m tS e rv ice R e qu e s t s

Q u ality C o n tro lB e lie f R e v is io n D yn am ic fo r In te r-L e v e l C o n s is te n c y

M S M T

M S M T

Pe di gre eIn form ati on

ES TI M A TES

ES TI M A TES

5) Pedigree5) Pedigree

• We define Pedigree as “an attachment to a massage or communication between nodes that includes any information necessary to the receiving node(s) such that the receiving node fusion processing maintains it’s formal and mathematical processing integrity”.

6) Ontologically-based Data Fusion Processes

• One important foundation toward achieving Interoperability and Shared Understanding, especially for Higher-Level Fusion states

• Ontological relationships as a basis for development of new Theoretical constructs for the “True” world

• Theory as a basis for Algorithm development in the observed world

What about a L2 Ontology?What about a L2 Ontology?• What is a “Situation”? --Not adequately specific

--No common understanding• How is it “Refined” --No metrics/quality measures• What kind of algorithmic process yields a “Situation Estimate”?• Llinas assertion: “Situation” is too coarse/abstract to engineer to—

MUST get more specificAlgorithm re

Thing-componentIn the

Real World

Situations in the Real World

PartitioningAnd

Labeling

(Analysis;Ontology—Sufficient SpecificityTo develop Theories)

Model (Theory) of Thing-component

In the Real World

Model (Theory) of Thing-component

In the Real World

Model (Theory) ofThing-component

In the Real World

Algorithm reThing-component

In the Real World

Algorithm reThing-component

In the Real World

Algorithm reThing-component

In the Observed World

Assumptions, Approximations,Application-needs

Nature of :•Aggregated Objects• Behaviors,• Events

Aggregated-object(“Convoy”)Tracking Algorithm

Task Reqmts

Informs, Bounds

Nature of, models of

Observational Processes

RealWorld Observed World

in the Application (Task) Context

UserWorld

What IS a “Convoy”?

Situations as inherent: an attack (situation) may be occurring even if the user’s task-at-hand has no interest in an attack state

Some set ofComponents, in some relationship= “Situation”

Observability

SummarySummary

• There is a clear need for expanding and enhancing the JDL Model to deal with and incorporate the effects of the various issues raised herein

• The Model has been an anchor-point for communication and understanding in the Fusion Community and has served us well but it needs contemplative review and a consensus-based modernization