DNI Blackbook: Semantic Data Management -1

8/2/2019 DNI Blackbook: Semantic Data Management -1

1/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO


2/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Blackbook2 is a J2EE server-based data integration

framework

Relies on open standards to promote robustness andinteroperability JENA, JUNG, Lucene, JAAS, D2RQ

Based on semantic web technologies RDF, RDF Schema, OWL, SPARQL Vocabulary agnostic

Provides a default web application interface, SOAP andRESTful interfaces

Blackbook2 is PL3 Appendix E certified (PL3+)

Overview

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO


3/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Architecture

Visualization Layer

Infrastructure Layer

Data Source Layer1 2 3


Data Source Layer


4/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Advanced Search

Workspace

A front-end Google-like userinterface allows analysts to easilyperform keyword and attributebased searches.

Advanced Search

Search


5/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Google-like Results

Network

Different ways to view the sameinformation. Network , for example,displays entities of different typesand their relationships to otherentities.


6/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Timeline

Multi-Panel

Different ways to view the sameinformation. Timeline , for example,displays entities chronologically

C SS // O O


7/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Google Map

Google Earth

Allows analysts to visualize geospatialcontent using Google-map and GoogleEarth.

UNCLASSIFIED//FOUO


8/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Analyst Notebook

Mediawiki

UNCLASSIFIED//FOUO


9/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Architecture

1 2 3

Visualization Layer


Data Source Layer

UNCLASSIFIED//FOUOUNCLASSIFIED//FOUO


10/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

The Semantic Web is the next generation of the current

web in which computers can interpret the meaning of theweb content because of explicit semantics provided inmarkup.

Semantic Web

Unicode URI

XML + namespace + xmlschema

RDF + RDF Schema

Ontology

Logic

Proof

Trust

DigitalSignature

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO


11/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Example 1: Inference

An analyst creates:

Buster

1) EntityBuster

Jennifer

2) Entity Jennifer

Nicholas

3) Entity Nicholas

An analyst makes the assertion:

mother-of

5) Jennifermother-ofNicholas

4) Busterhusband-ofJennifer

husband-of

Blackbook system can infer:

7) Nicholas child-ofJennifer

child-of

6) Jenniferwife-ofBuster

wife-of

9) Nicholas [step]child-ofBuster

[step]child-of

8) Buster[step]father-ofNicholas

[step]father-of

10) Busteris-genderMale

is-gender

Male

11) Jenniferis-genderFemale

Female

is-gender

UNCLASSIFIED//FOUO


12/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Example 2: Invalid Logic Assertion

Busterhusband-of

child-of

wife-of

[step]child-of

[step]father-of

is-gender

Male

An analyst creates:

1) EntityDon

Don


2) Jenniferbrother-ofDon

brother-of

Jennifer

mother-of

Female

is-gender

Nicholas


3) Invalid Assertion(Gender conflict)

UNCLASSIFIED//FOUO


13/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Buster

CandyBill

Example 3: Constraints & same-as

Mark William


has-brother

1)Busterhas-brotherBill

has-sister

2) Busterhas-sisterCandy

has-son

3) Candy has-son Mark

has-uncle

4) Mark has-uncle William

An analyst applies the constraint:

5)Busterhas-only-one-brotherBill

has-only-one-brother


6) William same-as Bill

same-assame-as 7) Bill same-as William

UNCLASSIFIED//FOUO


14/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Algorithms, Security, AKB

32

1

Algorithm plug-ins can be added

Security PL3+ / User Credentials CASPORT

Scattered CastlesDIAS

Analysts can store assertions into anAnalytic Knowledge Base (AKB)

UNCLASSIFIED//FOUO


15/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Workflow

Yahoo Pipes

Workflow allow analysts to define theorder of tasks, configure algorithmparameters, and batch processesconcurrently

UNCLASSIFIED//FOUO


16/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

fas

User Interface

Workspace

Workflow and Workspace allow analysts to definethe order of tasks, store them in private folders and/orshare them publicly with colleagues.

UNCLASSIFIED//FOUO


17/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

Architecture

Visualization Layer


Data Source Layer1 2 3


Data Source Layer

UNCLASSIFIED//FOUO


18/29

UNCLASSIFIED//FOUO

UNCLASSIFIED//FOUO

1

2

3

A

B

C

D

E

F

G

H

I

Original Datasource

AKB

Analyst Knowledge Base

E

G

K

C

D

J

A

B

C

D

E

F

G

H

I

Composite Knowledge

K

J

Composite Knowledge

UNCLASSIFIED//FOUO


19/29UNCLASSIFIED//FOUO

U

S

S

T

U

S

T

U

U

U

S

S

U

T

S

U

AKB

U

U

U

S

S

U

T

S

U

Original Datasource Analyst Knowledge Base Composite Knowledge

T T

S S

Composite Knowledge with Security

U

S

T

Unclassified

Secret

Top Secret

UNCLASSIFIED//FOUO



U

S

S

T

U

S

T

U

U

U

S

S

U

T

S

U

AKB

U

U

U

S

S

U

T

S

U

Original Datasource Analyst Knowledge Base Composite Knowledge

T T

S S

Composite Knowledge with Confidence

30%50%

100%10%

30%50%

100%

10%

UNCLASSIFIED//FOUO



fas

User Interface

Relationship Manager Entity Manager

Ontology Import

Allows analysts to specify the relation-ship between two or more entities Allows analysts to create entities ofdifferent types, and modify attributes

Allows analysts to upload their ownontology

UNCLASSIFIED//FOUO



Unstructured/Structured to RDF

Translation

Tran

slatio

n

Unstructured

Structured

RDFRDMS/XML

Unlike most applications, Blackbook performs queries on

data in RDF form, not relational form.

Sarin

Use ofAgent

Tokyo

Japan

Threatto Use

Unknown

Israel

Tel Aviv

Use ofAgent

NewYork

UnitedStates

urn:a911report:incident233

urn:sandia:organization65

urn:anubis:incident873

UNCLASSIFIED//FOUOUNCLASSIFIED//FOUO



Datasource Connectivity

UNCLASSIFIED//FOUO

UnstructuredText

TransformNet-OWL

D2RQ

MAP

LUCENE

TTT

RDF

TransformScript

RDF

TransformScript

RDFXXX

XML

RDBMS

XX

X

XML

JENA

MAP

MAP

Client

Stub

MAP

1

2

3

4

5

6Web-

Services

RDBMS

MAP

MAP

Currently, there are six avenues to connect Blackbook with

data; 1-3 requires offline translation to RDF, 4-5 uses

real-time translation using D2RQ, 6 uses secure web-services.

UNCLASSIFIED//FOUO



Blackbook and Alternate Stores

Hbase, Lucene,

Solr

UNCLASSIFIED//FOUO


25/29

UNCLASSIFIED//FOUO

Scalability using Hadoop

Scalable: Hadoop can reliably store and process petabytes.Economical: It distributes the data and processing across clusters of commonly availablecomputers. These clusters can number into the thousands of nodes.Efficient: By distributing the data, Hadoop can process it in parallel on the nodes where thedata is located. This makes it extremely rapid.Reliable: Hadoop automatically maintains multiple copies of data and automatically redeployscomputing tasks based on failures.

Hadoop implements MapReduce, using the Hadoop DistributedFile System (HDFS). MapReduce divides applications into manysmall blocks of work. HDFS creates multiple replicas of datablocks for reliability, placing them on compute nodes around thecluster. MapReduce can then process the data where it islocated.

UNCLASSIFIED//FOUO
http://hadoop.apache.org/


26/29

UNCLASSIFIED//FOUO

Blackbook and Wikis

Wikis

Wikis

2

UNCLASSIFIED//FOUO


27/29

UNCLASSIFIED//FOUO

Blackbook and Wikis

MySQL

Oracle

SOAP/REST

Browser

Wiki

ex) Intellipedia

Like browsers, Wikis can be a front-end toBlackbook. Wikis can also be a datasource. Wikiextensions can be utilized to enable Semantic and

Blackbook features.

Semantic extension

Blackbook extension

UNCLASSIFIED//FOUO


28/29

UNCLASSIFIED//FOUO

Revolutionize Technology Transfer

Research andDevelopment

BlackbookInfrastructure

T&E andDeployment

Research Government

Self-Addressed Stamped Envelope

A research product (red), such as a new and improvedalgorithm or visualization, can easily be transferred fromresearch to government using the Blackbookenvelope .

Improve Intelligence Analysis by Coordinated Exposition of MultipleData Sources Across Intelligence Community Agencies

UNCLASSIFIED//FOUO


29/29

Relational vs. Graph-based Systems

1 2 3


Data Source Layer

Most Systems

Relati

onal

1 2 3


Data Source LayerGraph

Blackbook2 is a JEE server-based RDF processor thatprovides an asynchronous interface to back-end

datasources.

Documents

DNI Blackbook: Semantic Data Management -1